Passive virtual reality

ABSTRACT

A method and apparatus for providing, in response to image signals originating in an object space, mixed image signals for providing nonuniform resolution images for stimulating simulated active percepts for passive perception by a viewer in an image space. The images have a highly detailed component which has its image content changed according to changes in the direction of a simulated observer&#39;s eye position in an object space. The images may be provided stereoscopically. The images may be provided at various apparent distances and the relationship between accommodation and convergence may be preserved. Audio waves for directionally simulating that which would be heard by the simulated observer may be provided.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a divisional of Ser. No. 09/072,202 filed May 4,1998 now U.S. Pat. No. 6,307,589, which itself is a divisional of Ser.No. 08/462,503 filed Jun. 5, 1995, now U.S. Pat. No. 5,748,382, whichitself is a divisional of Ser. No. 08/001,736 filed Jan. 7, 1993, nowU.S. Pat. No. 5,422,653, from which priority is claimed under 35 U.S.C.§ 120.

TECHNICAL FIELD

The present invention relates to the presentation of images and, moreparticularly, to the presentation of successive images.

BACKGROUND ART

Still photography, motion pictures and television were influenced by theway artists represented physical reality in paintings, as if through awindow. A highly detailed perspective image is provided, typicallywithin a rectangular frame. All provide highly detailed images whichinduce the viewer to cooperate with the cameraman's “vision” by assumingthe artificial perspective of the representation. The viewer is enabledto deliberately suspend disbelief that the images themselves are not areal object space. The degree to which the viewer is thus enabled isinfluenced not only by the image resolution but by the field of view. Itis usually thought desirable to increase both. For example, very highresolution commercial television standards have been formulated forincreasing image quality. Such approaches typically increase the numberof horizontal lines scanned to a number significantly greater thanpresent standards. Larger format movie film such as 70 mm has been usedto increase detail. Also, panoramic movies, e.g., “Cinerama” increasedthe field of view to increase realism. Various stereoscopic televisionapproaches have also been conceived or developed to increase realism.

All of these traditional media take a rather objective view of thephysical world. The image is framed by a window through which the viewercan gaze in any direction “into” a representation of an object space.Events are presented in both movies and television in a series ofdifferent action scenes in a story line which the viewer can observefrom a seemingly quasi-omniscient point of view. The viewer is led totake what appears to be a view of the world as it really is. Yet thechoice of image and its perspective is picked by the creator of theimage and the viewer actually assumes a passive role.

“Virtual reality,” in an electronic image context, goes even further inthe direction of increased realism but enables the viewer to take a moreactive role in selecting the image and even the perspective. It meansallowing a viewer's natural gestures, i.e., head and body movements, bymeans of a computer, to control the imaged surroundings, as if theviewer were seeing and even moving about in a real environment ofseeing, hearing and touching. Due to the myriad of possible actions ofthe viewer, a corresponding multiplicity of virtual activities needs tobe available for viewer choice. This would represent the ultimate inartificial experience.

But the creation of many possible scenarios for viewer selection createsa massive demand for electronic image storage space and there is alsothe problem of a disconcerting time lag between the viewer's action andthe response of the imaging system. These problems make this emergingtechnology hard to achieve using presently available hardware.

And it would seem impossible to carry out such a heightened artificialexperience using motion picture technology.

DISCLOSURE OF INVENTION

The object of the present invention is to provide a new method and meansof presenting images in succession.

According to a first aspect of the present invention, images simulativeof active percepts are provided for passive perception.

Simulated percepts, according to the present invention, permit a viewerto experience percepts as if inside the head of another person.

The simulated active percepts may be presented “live” or may be storedand retrieved from storage and presented for passive perception. In thecase of stored simulated active percepts, since there is only one set ofimages to store, the memory problem of the prior art is solved.Similarly, for the “live” case, since the simulated active percept isprovided as created there is no storage requirement at all. Moreover, byproviding simulated active percepts for passive perception, there is nolonger any time lag problem. Since the simulated active percepts inducethe viewer to emulate those physical actions which would've created thesimulated active percepts, the hardware need not be faster than or asfast as the viewer. In fact, it may be much slower. Although the vieweris relegated to a rather passive role, the novelty and richness of thevirtual reality experience more than compensates in opening a whole newworld of opportunity for representing reality.

In further accord with this aspect of the present invention, the imagessimulative of active percepts are nonuniform images. The human visualapparatus does not resolve images uniformly, since the human eye'sretina converts optical images cast upon it with nonuniform resolution.The portion of the image about the axis of the observer's gaze is caston the fovea centralis, for high detail perception of the imageinformation, while peripheral image areas are cast on the remainder ofthe retina for perception with less detail. Thus, there is a fundamentalmismatch between the traditional manner of pictorially representing,with uniform resolution, the objects which constitute physical realityand the manner in which we actually sense it.

Although the visual apparatus senses the details of objectsnonuniformly, human perception is of objects having uniform resolution.Such perception is formed by integrating a series of fixations overtime. To complicate matters, any given observer's point of view withinan object space is not generally kept stable at all. As the observermoves about with three translational degrees of freedom in the objectspace his head may continually be changing its orientation as permittedby its three rotational degrees of freedom. At the same time, theobserver's eyes, with three rotational degrees of freedom of their own,are continually fixating on different objects within that space. Thereis no experience of disorientation because the observer senses that theobject space is stable with respect to his head and eyes moving aboutunder self-impetus. A sense of balance is lent by the mechanism of theinner ear. In addition, visual clues, including nasal and eye orbitalshadows indicate the present orientation of the head with respect tothat of the eyes and with respect to the object space.

In the case of paintings and still photography, the point of view is ofcourse unchanging. But, even for movies and TV, a given point of view,once acquired, is kept rather stable and is generally changed only veryslowly to avoid viewer disorientation. Most have experienced somediscomfort when viewing a stable movie or TV image which is suddenlymoved quickly in a lateral direction or which suddenly tilts from itsnormally horizontal orientation. This effect is due to the upset of astable, uniform image of an object space which purports to represent orsubstitute for physical reality. To upset the frame is akin, accordingto the perception of the viewer, to upsetting the object space itself.Hence, the discomfort.

Thus, there is also a fundamental mismatch between the traditionaltechnique of static or quasi-static image presentation used in the movieand consumer TV media, described above, and the manner in which imagesare actually gathered and experienced by an observer using his humanvisual apparatus. I.e., where the observer may quickly move about in anobject space, in a process of frequently changing his point of view andhis gaze in order to acquire visual information from all directionsconcerning his surroundings.

In keeping with these teachings, in further accord with the first aspectof the present invention, images are presented to simulate movement ofthe observer's eye's orbital shadow, i.e., the overall direction of viewwith respect to the object space for stimulating the passive viewer toexecute analogous head movements.

In further keeping with these teachings, in still further accord withthe first aspect of the present invention, nonuniform resolution imagesare presented which simulate ductions of a simulated eye with respect tothe simulated eye's orbit for stimulating a passive viewer's eye toexecute analogous ductions and which may, at the same time, orindependently, simulate movement of the orbital shadow, i.e., the pointor field of view with respect to the object space, for stimulating thepassive viewer to execute analogous head movements.

In further accord with this first aspect of the present invention,successive mixed optical images of an object space are provided in animage space for a viewer's eye or for the viewer's visual apparatus,each mixed optical image having a highly detailed component and a lesserdetailed component; the image content of selected successive mixedoptical images being changed according to changes in the direction ofthe simulated eye's visual axis in the object space; the highly detailedcomponent for being cast on the fovea of the retina of at least one ofthe viewer's eyes and the lesser detailed component for being cast on anonfoveal part of the retina of one or the other (not necessarily thesame one that the highly detailed component is cast on) of the viewer'seyes such that the relative direction of the visual axis of at least oneof the viewer's eyes with respect to the image space may analogouslyfollow the relative direction of the simulated eye's visual axis withrespect to the object space. The mixed images may be made in any numberof different ways. For example, one or both eyes may be presented withmixed images made up of one or more wide field of view fields or framesof low detail followed at a selected point in time by one or more highdetail fields or frames over a narrow field of view. Or, one eye may bepresented with only narrow field of view high detail images while theother is presented with only low detail images covering a wide field ofview. The manner of constructing and unifying the mixed images into acoherent presentation presents the designer with almost limitlesspossibilities and variations. In such a case it is difficult to describeall possible future implementations of the claimed method and apparatusor all ways of constructing “mixed images.” Even though it is difficultor even impossible to describe all such variations, an attempt is madein this disclosure to briefly describe several such variations in orderto show the breadth of possible approaches and that therefore all suchvariations, elaborations, take-offs, interpretations, etc., of theteachings hereof, and which are fairly within the scope of the claims,are covered thereby. As a consequence, according to the presentinvention, whatever the method selected for constructing “mixed images,”at least one of the foveae of a viewer may be stimulated by high detailimages having a narrow field of view and at least one of the retinae ofthe viewer may be stimulated by low detail images having a wide field ofview, the total effect being unified perception of nonuniform resolutionimages. Of course, the simplest approach is to present both a highlydetailed image and a lesser detailed image to one or both eyes. Anothersimplification is to limit the number of degrees of freedom represented,e.g., by only simulating two of the three degrees of rotational freedomof the eye, i.e., by omitting the simulation of torsion. Of course, onlyone degree of freedom, e.g., horizontal, may be acceptable for manypurposes. Or, the translational or head movement degrees of freedomcould be limited or even omitted in some approaches.

In still further accord with this first aspect of the present invention,the highly detailed component is mobile with respect to the lesserdetailed component to simulate ductions of the simulated eye withrespect to the simulated eye's orbit, the lesser detailed componentencompassing a field of view which changes to simulate movement of theorbit's head with respect to the object space. In other words, thehighly detailed component moves about to simulate at least two of therotational degrees of freedom of the eye (ductions) while the lesserdetailed component changes its point of view to simulate the rotationaldegrees of freedom of the head. Simulation of translational degrees offreedom is provided by translating the point of view of the lesserdetailed component.

Alternatively, the highly detailed component is immobile with respect tothe lesser detailed component wherein the highly and lesser detailedcomponents are jointly mobile to simulate eye ductions with respect tothe simulated eye's orbit. In that case, the lesser detailed componentof the successive mixed optical images encompasses a field of view whichchanges to simulate the simulated eye's ductions with respect to theorbit's head and which also changes to simulate movement of the headwith respect to the object space. This alternative may be effectivelycarried out in a real object space, e.g., by mounting a camera having anonlinear lens (such as disclosed in U.S. Pat. No. 3,953,111) on a twodegree of freedom platform for simulating two of the three degrees offreedom of the eye in its socket (omitting torsions) and mounting theplatform on another platform having three rotational degrees of freedom,e.g., a cameraman's head.

In still further accord with this first aspect of the present invention,the successive mixed optical images are provided for panoramicpresentation to a viewer's eye.

In still further accord with this first aspect of the present invention,the successive mixed optical images are provided at various apparentdistances such that a viewer's eye may accommodate to focus on thesuccessive mixed optical images at the various apparent distances.

In still further accord with this first aspect of the present invention,the various apparent distances are selected in such a way as to preservea normal relationship between accommodation and distance.

It should be understood that the provision of mixed images forperception by a passive viewer, according to the first aspect of thepresent invention, separately encompasses the two fundamental processes(and the separate and distinct means for carrying them out) normallyassociated with the commercial exploitation of motion picture andtelevision productions. I.e., the unitarily claimed invention coverseither and both the creation or presentation of images:

(i) in response to an image signal, e.g., and without limitation, anoptical image of an actual object space, the creation of successivemixed image signals for immediate presentation or for storage for laterretrieval, and

(ii) in response to an image signal, e.g., to retrieved mixed imagesignals of the type provided “live” or stored as described in (i) above,the presentation of mixed optical images for presentation to a passiveviewer.

This further teaching of the first aspect of the present invention maybe stated as the provision of mixed image signals, which may beelectrical, optical, or generally electromagnetic, in response to animage signal which may or may not be mixed and which may also begenerally electromagnetic of any type.

According to a second aspect of the present invention, additionalsuccessive mixed optical images of the object space are presented in theimage space for presentation to the viewer's visual apparatus, eachadditional mixed optical image having a highly detailed component and alesser detailed component, the image content of selected additionalsuccessive mixed optical images being changed according to changes inthe direction of an additional simulated eye's visual axis in the objectspace; the highly detailed component for being cast on the fovea of theretina of at least one of the viewer's eyes and the lesser detailedcomponent for being cast on the retina of at least one of the viewer'seyes such that the relative direction of the visual axis of at least oneof the viewer's eyes with respect to the image space may analogouslyfollow the relative direction of the additional simulated eye's visualaxis with respect to the object space. The stimulation of a viewer'seyes with images from different perspectives produces stereopsis. Again,the mixed images may be constructed in any number of different ways. Forexample, a small, highly detailed image from one perspective might bepresented to one eye for its fovea and a large, lesser detailed imagemight be presented from another perspective to the other eye for itsretina. Or, each eye may be presented with both highly and lesserdetailed image areas from different perspectives, i.e., both the highlyand lesser detailed image areas in each eye being from the sameperspective. Or, only one eye may be provided with both highly andlesser detailed image areas, with the other having only a lesser orhighly detailed image area presented to it. Other combinations are ofcourse possible, and well within the scope of the present invention. Andit should also be understood that not only may the method ofconstructing the binocular images be selected from a wide variety ofdifferent approaches, but also the manner of constructing the individualimages or image portions may also be selected from a wide variety ofdifferent approaches. To take but a single simple example, for the casewhere both highly detailed and lesser detailed images are presented fromdifferent perspectives to each eye separately, the lesser detailed imagepresented to the left eye from a left perspective, and similarly to theright eye from a right perspective, are constructed from a horizontalscanning scheme similar to that used in conventional commercialtelevision in which two separate “fields” are interlaced to form asingle “frame” except that a small void may be left unscanned forfill-in by a high detail scan, also taken from the appropriate left andright perspectives. Each void may be filled in by the high detail scansubsequent to each of the field scans. However, it will be understoodthat, using such an approach, it could be filled in concurrentlythereto, or subsequent to the frame, or even subsequent to a field orframe for the other eye. Needless to say, a great many other approachesmay be practicable and be well within the scope of the present inventionwhich fairly covers all approaches to the construction of mixed imagesconstructed for passive viewing in furtherance of the object of thepresent invention.

In still further accord with this second aspect of the presentinvention, the successive mixed optical images and the additionalsuccessive mixed optical images are presented at various apparentdistances such that the viewer's eyes may accommodate to focus on themixed successive images and the additional successive images at thevarious apparent distances. Passive viewer versions about the imagespace at the various apparent distances are thereby induced.

In still further accord with this second aspect of the presentinvention, the various apparent distances are selected so as to preservea normal relationship between accommodation and convergence for theviewer. In this way, the viewer's induced passive versions at thevarious depths within the image space correspond closely to activeversions by which the viewer would normally acquire visual informationin real object spaces.

In still further accord with this second aspect of the presentinvention, the successive mixed optical images are provided from atleast one image source for presentation to at least one of the viewer'seyes and wherein the additional successive mixed optical images areprovided from at least another image source for presentation to at leastone of the viewer's eyes.

Alternatively, in still further accord with this second aspect of thepresent invention, the successive mixed optical images and theadditional successive mixed optical images are provided from a singleimage source and wherein the successive mixed optical images are forpresentation to at least one of the viewer's eyes and wherein theadditional successive mixed optical images are for presentation to atleast one of the observer's eyes.

According to a third aspect of the present invention, audio waves areprovided from a plurality of sources for the passive viewer forsimulating audio waves which might be experienced by a pair of simulatedears located on opposite sides of the simulated eyes' head. One approachis to provide two or more speakers equally spaced about a horizontalequator around the passive viewer's head and to provide two or moreadditional speakers, equally spaced about a vertical equator at rightangles to the horizontal equator. In this way, sounds may be simulatedcoming from any direction by varying the intensity of the sound wavesproduced by the various speakers.

The present invention provides a new approach to the presentation ofsuccessive mixed optical images of an object space to a passive viewerin an image space. Passive perception of simulated active perceptsseemingly puts the viewer inside the head of another person. The furtherapproach of providing mixed images having highly detailed and lesserdetailed components is for inducing a more subjective “reperception” ofthe simulated active percepts albeit passively. A few words about themeanings of some of the more important words used herein are in order atthis point.

The words “object space” mean a real or imaginary space represented bysuccessive images.

The word “observer” means a real or hypothetical person in an objectspace.

The word “viewer” means a real person in an image space.

The words “image space” mean a space used to provide images.

By “percept” is meant a real or hypothetical visual impression in arespective real or hypothetical visual apparatus.

A “active percept” refers to a percept attributable to body, head or eyemovements of an observer.

The words “simulated active percept” denote an image for stimulating apassive viewer's visual apparatus.

The words “visual apparatus” are used to mean one or both eyes, opticnerves, tracts, cerebral cortex or associated parts thereof in anycombination.

A “passive viewer” means one whose body, head or visual axis mayanalogously follow the body, head or visual axis of an observer byfollowing simulated active percepts.

The words “active viewer” are meant to describe one whose activity orthe effects thereof are monitored to provide a control signal whichaffects the information content of an image presented thereto. Forexample, in a remote viewing invention disclosed by Holmes in U.S. Pat.No. 3,507,988, a viewer provides the impetus for eye movements and bymonitoring the viewer's eye, the scanning of a remote camera may becontrolled according to the viewer's eye movements.

A “mixed image” comprises one or more images of simulated activepercepts having areas of greater and lesser resolution together beingsimulative of foveal resolution. For example, such may comprise bothhighly detailed and lesser detailed portions or, alternatively, in thesense that individual images may individually be of uniform butdifferent resolutions, i.e., may be either highly or lesser detailed,but together such images may be interleaved in some convenient manner,not necessarily alternately, to form a series of images which have thesame effect as a series of images with different resolutions in each.

The words “image signal” means an electromagnetic manifestation, whichmay be conditioned or not, of an object space.

A “mixed” image signal is an image signal conditioned so as to produce amixed image.

A “virtual eye” means an eye of a theoretical visual apparatus locatedin an object space. Although a cameraman whose eye or eyes are monitoredfor controlling the scanning of a camera or pair of helmet mountedcameras mounted on his head, according to one embodiment of the presentinvention, might, in combination and in essence, approximate theintended definition, the meaning is slightly more precise. In such acontext, the cameraman's monitored eye is not in the same exact positionas the camera's “eye,” i.e., optics and light sensitive surface so thereis a parallax problem. So the theoretical visual apparatus in thatcontext is somewhere in between. In that situation, for example, thedefined words mean the theoretical visual apparatus that would, if realand located in the object space, stimulate essentially the same perceptssimulated in the image space by way of optical images presented to apassive viewer. For simulated active percepts made by computer oranimation the meaning is similar.

The present invention thus provides a new method and means of presentingimages of an object space for viewing by a passive viewer as if throughthe eyes of an active observer actively looking about within such anobject space. Stated another way, the present invention teaches thepresentation of images simulative of percepts of optical images of anobject space as if cast on an active simulated eye's retina forstimulating similar percepts in a passive viewer. Stated negatively, theteachings of the present invention show not how to represent an objectspace for active perception, but how to present images simulative ofactive percepts of an object space for passive perception.

Thus, the present invention is distinguished from the accepted and timehonored approach in the present state of the art of passive viewing, inwhich the object is to objectively portray an object space or, morerecently, to improve the objective realism of the object space (withoutchanging the basic approach) by increasing the field of view andimproving image resolution. The present invention, on the contrary,teaches the depiction of a real or simulated observer's subjectiveexperience of an object space for “re-experience” or “reperception” byan actual, but passive viewer. Increased realism in the usual sense iseschewed but, paradoxically, the effect in the passive viewer is anincreased sense of reality, as experienced through the “eyes” of“another.” Thus, the present invention provides a new way of passive,not active, viewing.

The state of the art approach to motion picture and televisionproductions is to provide a view of reality through a quasi-omniscient“eye” which purports to see things as they are in themselves, while theapproach of the present invention is to provide a new way of looking atthings, as seen by a selected observer.

Moreover, the approach for presenting images, according to the presentinvention is, in effect, backwards, upside down and the reverse of thenew technology of virtual reality. A passive viewer, according to thepresent invention, follows simulated active percepts.

The present invention may be used in a wide variety of applicationsincluding entertainment, education and others in which passive viewingis appropriate. The manner of presentation is so realistic and such aradical departure from present approaches that viewers are better ableto suspend disbelief that the images are not real. In fact, viewersactually perceive an object space by way of simulation of the humanvisual process, as if through the eyes of another person. Thetraditional approach of placing the passive viewer in the position of anobjective, quasi-omniscient observer is thus replaced by placing him inthe position of “reperceiving” the perceptions of a subjective observer.

The simulated active percepts may be presented to a screen for viewingby one or more passive viewers at once such as is shown in U.S. Pat. No.4,515,450 & U.S. Pat. No. 4,427,274 or PCT Patent WO 86/01310 inconjunction with, e.g., a pair of light shutter or polarizer glassessuch as shown in U.S. Pat. No. 4,424,529, or may be provided via imagesources in a helmet for mounting on a passive viewer's head in anapproach suggested by U.S. Pat. Nos. 4,636,866, or 4,310,849, or manyother possible presentation approaches.

These and other objects, features and advantages of the presentinvention will become more apparent in light of a detailed descriptionof a best mode embodiment thereof which follows, as illustrated in theaccompanying drawing.

BRIEF DESCRIPTION OF THE DRAWING

FIG. 1 shows conversion of an image signal to a mixed image signal,according to the present invention;

FIG. 1A shows a camera embodiment of the present invention whereinuniform resolution optical images are electronically converted tononuniformly resolved electrical signals;

FIG. 1B shows another camera embodiment of the present invention whereinnonuniform resolution optical images are converted to similarlynonuniformly resolved electrical signals;

FIG. 1C shows a display embodiment of the present invention wherein amixed image electrical signal, as provided by a camera embodiment of thepresent invention as in FIG. 1 or 2, for presentation of simulatedpercepts according to the present invention;

FIG. 2 is an illustration of an object space and of an image space, boththe object space and the image space containing an apparatus, accordingto the present invention;

FIG. 3 illustrates a modification of the apparatus in the image space ofFIG. 2 in which means are provided for changing the apparent distancesof the images presented, according to the present invention;

FIG. 4 is an illustration of two eye position monitors and a controlwhich may be used, according to the present invention, to determine thedistance to objects of regard;

FIGS. 5( a)-(d) are illustrations of various methods of providing imagesat various apparent distances, according to the present invention;

FIG. 6 shows a positional actuator for moving a lens, according to thepresent invention;

FIG. 7( a) is a simplified illustration of a portion of a displaysurface, according to the present invention

FIG. 7( b) is an illustration of the present invention as carried out ona display surface, for example, having addressable pixels;

FIGS. 8-12 are illustrations of various scanning techniques forconstructing mixed images, according to the present invention, withoutlimitation;

FIG. 13 shows how FIGS. 13( b) and 13(c) fit together and it also showshow FIGS. 13( d) and 13(e) fit together.

FIG. 13( a) is a block diagram of an apparatus usable for controllingthe construction of mixed images in a display, according to the presentinvention;

FIG. 13( b) is a block diagram of a display controlled by the apparatusof FIG. 13( a), according to the present invention;

FIG. 13( c) is a block diagram of a solid state camera for providing avideo signal

FIG. 13( d) shows a portion of a solid state video camera such as thecamera 66 of FIG. 2.

FIG. 13( d) fits together with FIG. 13( c) in the manner shown in FIG.13 to show the solid state video camera having mixed image informationencoded therein for display on a display apparatus such as that of FIG.18, according to the present invention;

FIG. 14 is an illustration of a composite video waveform such as may,without limitation, be provided by the camera of FIG. 1, according tothe present invention;

FIG. 15 is an illustration of a means and method of constructing andpresenting panoramic mixed images, according to the present invention;

FIG. 16 is a block diagram of a means and method of constructing andpresenting stereoscopic mixed images, according to the presentinvention;

FIG. 16( a) is an illustration of a means and method of providingmultiphonic sound from various directions to a passive viewer, accordingto the present invention;

FIG. 17 is a block diagram of a means and method for presentingstereoscopic mixed images at various apparent distances, according tothe present invention;

FIG. 18( a) is an illustration of the manner in which the monocularmixed image fields of view of a pair of virtual eyes are perceivedbinocularly by a passive viewer, according to the present invention;

FIG. 18( b) is an illustration of the human binocular horizontal fieldas seen from above;

FIG. 18( c) is an illustration of a left monocular field of view of avirtual eye, according to the present invention;

FIG. 18( d) is an illustration of a right monocular field of view of avirtual eye, according to the present invention;

FIG. 19 is an illustration of the relationship between accommodation andconvergence for two normal interocular distances;

FIG. 20 is an illustration of a Brewster-Holmes stereoscope adapted,according to the present invention, for presenting successive mixedimages in a manner which provides for a substantially constantaccommodation/convergence ratio;

FIG. 21 is an illustration of a Risley prism as may be used, accordingto the present invention;

FIG. 22 is an illustration of FIG. 18( c) & 18(d) except showingoversized matrices, according to the present invention;

FIG. 23 is an illustration of a helmet embodiment of the presentinvention for use in an image space by a passive viewer;

FIG. 24 is a top view of the helmet of FIG. 23; and

FIG. 25 is an alternate top view of the helmet of FIG. 23.

FIGS. 26-35 are illustrations of a variable magnification lens,according to the present invention;

BEST MODE FOR CARRYING OUT THE INVENTION

FIG. 1 shows a mixed image signal on a line 1 provided after aconversion step or device 2, in response to an image signal 3 which mayitself be a mixed image signal. Image capture methods and embodiments ofthe present invention are shown in FIGS. 1A and 1B and an imagepresentation or display method and embodiment is shown in FIG. 1C. Inthe practice of the invention as illustrated in FIGS. 1A& 1B, anapparatus or step 2A and 2B are similar to the device or step 2 ofFIG. 1. These steps or embodiments 2A, 2B may include uniform resolutionoptical conversion or optics 4A or nonuniform resolution conversion oroptics 4B and, in addition, a light sensitive surface 5A and 5B,altogether in an object space; the image signal 3 may be an opticalimage or light signal 3A, 3B provided to a light sensitive surface inthe camera. The light sensitive surface is analogous to a retina of anobserver in the object space. The light sensitive surface in the cameramay be used to convert a uniform resolution light image signal to anencoded image signal of nonuniform resolution as in FIG. 1A. Or, it maybe used to convert a nonuniform resolution light image signal to anencoded image signal of similar nonuniform resolution as in FIG. 1B. Ineither event, for the camera embodiments, the mixed image signal fromthe light sensitive surface is an encoded nonuniform resolution or“mixed” resolution image signal. It would, but need not, typically takethe form of an electrical signal having the mixed images encodedtherein. There could of course instead be an optical to optical signalconversion but the present state of the art is not yet at that stage. Ina display embodiment of the invention as shown in FIG. 1C, theconversion step or device 2 may be carried out by or be itself a lightemissive surface 2C in an image space and the image signal maytypically, but need not, be an electrical mixed image signal similar to,or the same as, that provided by the camera embodiment. Again, ofcourse, there may be an optical to optical conversion instead. Thedisplay embodiment is for providing, in response to such a mixed imagesignal, an optical mixed image signal which is an actual light image forstimulating simulated percepts in a viewer's eye, according to thepresent invention, wherein a viewer may virtually experience visualreality, as if in the object space and inside the observer's head,passively following the activity of the observer's visual apparatus.

FIG. 2 is an illustration of an object space 20 and of an image space 22respectively containing an apparatus 24 and an apparatus 26, eachaccording to the present invention. The apparatus 24 is located in theobject space and is responsive to image signals, e.g., reflected fromobjects in the object space and provides mixed image signals fortransmission to a storage medium or, as shown without limitation,directly to the image space. The apparatus 26 is located for use in theimage space 22 and is responsive to mixed image signals from the storagemedium or, as shown, directly from the apparatus 24 in the object spacefor providing successive mixed optical images, also called mixed imagesignals, as indicated by an image signal on a line 28, to a passiveviewer 30. As previously defined, “image signal” can mean anyelectromagnetic manifestation, whether conditioned or not and thus mayencompass mere light rays reflected from objects such as the imagesignal on line 32 of FIG. 2, as well as conditioned electromagneticmanifestations such as the image signals on lines 34, 36, 38, 40, or 42.Additionally, the image signal on line 28 need not be provided directly,as shown, but may be reflected from a surface or may be transmittedthrough a medium. As defined above in the disclosure of inventionsection, and as explained in more detail below, the mixed optical imagesignals have highly detailed and lesser detailed components mixedtogether. The highly detailed component is designed to be cast on thefovea of a passive viewer's eye 42 and the lesser detailed component onthe remainder of the passive viewer's retina. The mixing of highlydetailed and lesser detailed image components lends a certaindirectionality to the images presented which induces the passive viewerto shift his gaze according to changes in the position of the highlydetailed component with respect to the object space imaged or,alternatively, with respect to the lesser detailed component.

The apparatus 26 of FIG. 2 comprises a control 44, responsive to a videosignal on a line 40 from a receiver 48, for decoding image informationfrom the video signal and providing the image information in a signalformat suitable for presentation by an image display or source 50, asindicated by a signal line 52. A control signal on a line 54 changes theportion of the object space represented in detail at any given time,i.e., changes the image content of selected successive images and hencethe position of the highly detailed component with respect to the lesserdetailed component or, alternatively, with respect to the apparentposition of the object space imaged by the lesser detailed component,according to changes in the direction of a simulated active eye's visualaxis in the object space 20, as decoded from the signal on line 40.Although the signals on lines 40, 52, 54, and many other signals hereinare shown as single lines, it should be understood that there may bemore than one signal associated with each.

A simulated active eye (not shown) is postulated in the object space,for the teaching purpose of broadly describing an associated simulatedvisual axis actively moving about in the object space, which the passiveviewer may analogously follow with his own visual axis 56 in the imagespace; the images are presented to the passive viewer's eye 42 in theimage space from a perspective which analogously coincides with that ofthe simulated active eye in the object space. In other words, theposition of the center of rotation of the simulated active eye in theobject space with respect to the objects “observed” and imaged,corresponds to the position of the center of rotation of the passiveviewer's eye in the image space with respect to the images of objectspresented thereto. The passive viewer's visual axis analogously followsthat of the simulated eye because his visual instinct is naturally drawnto cause the high detail part of the image to be cast on his eye'sfovea. Thus, the passive viewer's eye 42 mimics such a simulated activeeye. An apparatus 58, according to the present invention, shown in theobject space of FIG. 2, together with a cameraman 60, approximatelyprovide the function of such a simulated eye. The cameraman's eye 62provides a moving visual axis 64 and a video camera 66 provides a meansof capturing images. The camera, for some applications, is of theminiature type which is mounted in a helmet for the cameraman. The eye62 is shown directed at an object 68 illuminated by a light source 70,not necessarily localized in the object space. Of course, the objectspace will have numerous such objects which will successively be engagedby the cameraman's visual attention, over time; merely one such instanceis shown. An incident ray 72 is shown impinging on the object 68 and areflected ray 32 is incident on a lens 74 in the video camera. Numeroussimilar rays (not shown) combine to form an image of the object in thecamera. The cameraman's head may be in close proximity to the apparatus58 so that the axis 64 in the eye's principal position is more or lessparallel to a line normal to the light sensitive surface in the camera.The closer the camera can be placed to the eye 62, the lesser will bethe parallax effect caused by the distance therebetween and the closerwill the apparatus approximate the postulated simulated eye. An eyeposition monitor 76 monitors the position of the cameraman's eye 62 bymeans, for example, of an oculometer, which directs an invisible beam 78of infrared radiation onto the eye 62 where it is reflected back fordetection. An eye position signal, indicative of the direction of thecameraman's visual axis in the object space, is provided on a line 80 toa control 82. The eye position signal 80 is used by the control 82 tocontrol the portion of the object space which will be imaged in a highlydetailed manner in the video camera 66 by means of one or more controlsignals on a line 84. Numerous eye tracking devices, other thanoculometers, are generally known in the art of eye tracking and may befound, without limitation, in the U.S. patent literature in class351/subclasses 6 and 7. For example, a basic oculometer, such as wasdescribed above, is disclosed in U.S. Pat. No. 3,462,604. An example ofanother type of eye tracker, based on the detection of Purkinje images,is disclosed in U.S. Pat. No. 3,712,716. Still another example of a typeof eye tracker is disclosed in U.S. Pat. No. 4,561,448, based onelectro-oculography. These are examples only and should not be taken aslimiting the choice of eye trackers or eye tracking methods, as any typeof eye tracking method or apparatus capable of tracking the position ofthe visual axis of the cameraman's eye is encompassed by the monitor 76.

The camera 66 in turn provides image information over a signal line 86back to the control 82, where the information is encoded and provided asa video signal on a line 88 to a transmitter 90. It is necessary topoint out that the encoding of the video image information into acomposite video signal or component signals could as easily be done inthe camera 66 itself, rather than the control 82. Similarly, the controlfunction and/or the eye tracking function may be effected within thecamera structure also. The transmitter provides a video transmissionsignal on a transmitting antenna line 92 for transmission to the imagespace via a broadcast signal line 94. The broadcast signal is detectedby a receiving antenna 96 which provides the transmitted video signal tothe receiver 48.

It should be pointed out that the video signal ultimately provided onthe line 40 to the apparatus 26 may be formed in a manner quitedifferent from that shown in FIG. 2. For example, the object space neednot be real and the image signals may be formed by means of traditionalor computer animation, without using a cameraman and without monitoringany of his eyes. In that case, the signal on line 40 may not containimage information per se but merely control signals, e.g., for a motionpicture camera or for light valves. Similarly, the mixed image signalsmay be constructed by means of a computer. Furthermore, the images neednot be broadcast. They could be provided in other ways, e.g., via cableor from a video tape by means, for example, of a video cassette recorder(VCR). Thus, they need not be generated and viewed at the same time, asin FIG. 2, but may instead be recorded using a recording medium such asvideo tape for storing the video signals for later display, e.g., usinga video playback device. Thus it will be understood that the apparatus58 of FIG. 2 is merely shown for the purpose of illustrating one way inwhich a simulated active eye's images may be constructed and deliveredfor viewing by a passive eye. Other ways are of course within the scopeof the present invention.

FIG. 3 illustrates the apparatus 26 of FIG. 2 modified, in a moresophisticated embodiment, to include means 98 for presenting the mixedoptical images 28 at various apparent distances, in response to adistance signal on a line 100, indicative of the distance from thecameraman's eye 62 to the object 68. The means 98 may, withoutlimitation, be a variable magnification lens or combination of lenses,such as any of the various types disclosed in the patent literature inClass 350/Subclass 419. The fact that FIG. 3 shows the optical imagespassing through the means 98 does not exclude, as an alternative,reflection of the images off of a screen for passive viewing. Thus, themeans 98 may, also without limitation, be a variable focus mirror, suchas disclosed in U.S. Pat. No. 3,493,290. Another example of such amirror may be found in an article by Eric G. Rawson entitled “3-DComputer-Generated Movies Using a Varifocal Mirror,” Applied Optics,August 1968, Vol. 7, No. 8, pp. 1505-12. Another approach would be touse a modified refractive state controller, such as disclosed in U.S.Pat. No. 4,190,332; one of the necessary modifications to a device ofthat kind would be to remove the means for inducing artificialoscillations to attract the patient's attention. However, the idea ofmoving the screen is an alternative particularly advantageous for astereoscopic embodiment of the present invention as described below.

It is often said (see, for example, U.S. Pat. No. 4,048,653, column 3,lines 9-24) that the brain is relatively insensitive to eye focus as aclue to the distance of objects seen and that one may therefore presentan image at any apparent distance, using the size and perspective ofobjects as clues, although the viewer's eye remains fixedly focused at ascreen, for example. While this is certainly true, especially to theextent that it is consistent with the broad object of the presentinvention, it is nevertheless a corollary of the teachings hereof toshow how to present images at distances or apparent distances whichcorrespond to the distances a viewer would normally experience using hisaccommodative faculty as an observer in a real object space. In otherwords, the distance or apparent distance of the image presented ischanged in a manner which is consistent with the relation between thedegree of accommodation in the normal human eye and the distance of realobjects viewed. For each of the distances viewable by a normal eyebetween infinity and the nearest point of distinct vision there will bea corresponding state of accommodation. According to the presentinvention, this relationship is preserved by presenting images atapparent distances according to that relationship or one akin to it.

The distance signal on the line 100 is derived from distance informationencoded in the video signal on the line 40. The distance informationmay, without limitation, be obtained in the object space by monitoringboth of the cameraman's eyes. For example, in FIG. 4 there areillustrated two eye position monitors 102, 104, one for each of thecameraman's eyes 106, 108. The control 82 is responsive to a pair ofsignals on lines 110, 112 indicative, respectively, of the separatedirections 113 a, 113 b of the left and right eyes 106, 108 of thecameraman 60. A determination is made by the control 82 of the distancebetween the cameraman 60 and the object 68 based on the angle betweenthe left and right visual axes at their point of intersection and theknown interocular distance. (The control 82 may provide a lens controlsignal (not shown) for controlling the magnification of the lens 74 butsuch is not essential). The refractive state of the eyes may bemonitored using an objective refractor such as the 6600 Auto-Refractormade by Acuity Systems and described in U.S. Pat. No. 4,190,332. In sucha case, the other eye would not necessarily have to be monitored sincethe assumption could be made that both eyes approximately are at thesame accommodation level. There are of course many other rangefindingtechniques which may be usefully employed for the same purpose.

FIGS. 5( a) & 5(b) are illustrations of the principles upon which avariable magnification stationary lens or lens combination 94 mayoperate in functioning as the means 98 of FIG. 3 for presenting imagesat various apparent distances for a passive viewer's eye 42. FIG. 5( c)illustrates a fixed focal length movable lens or lens combination 116for similarly functioning as the means 98 of FIG. 3. In all of theillustrations of FIG. 5( a)-(c), a passive viewer's eye 42 is shown withits visual axis perpendicular to an image plane 118 which might be, forexample, an image projected on the inside of a hemispherical screen orprovided directly (without reflection) from a CRT, electroluminescent,gas plasma, liquid crystal, or the like display. The distance (d), ineach illustration, between the image plane 118 and the vertex of theeye's cornea is the same. The size of an image 120 is also the same foreach of the illustrations of FIG. 5. Only the apparent size of the image120 changes by reason of the different sizes of its image cast on theeye's retina.

In FIG. 5( a), the eye 42 is shown with its crystalline lens 122 in theaccommodated state, i.e., apparently for viewing an “object” 120 closeup. A variable magnification lens 114 is interposed between the eye 42and the image 120 for changing the apparent size (and hence the apparentdistance) of an image 124 cast on the eye's retina 126. A bundle of rays128 is illustrated emerging from a point 130 at one end of the image 120and their paths through the lenses 114,122 may be traced all the way tothe point's image 132 on the retina 126. Similar paths could be drawn toillustrate the retinal imaging of all the other points of the image 120.

In FIG. 5( b), the eye 42 is shown with its crystalline lens 122 in theunaccommodated state, i.e., apparently for viewing a distant “object”120. The variable magnification lens 114 is now shown schematically witha shorter focal length than it had in FIG. 5( a). This results in asmaller retinal image 134 which can only be kept in focus on the retinaby changing the shape of the crystalline lens 122, as shown. The sizeand thus the apparent distance of the image is reduced and a passiveviewer viewing such images experiences the sensation of viewing adistant object.

In FIG. 5( c), the eye 42 is again shown in the unaccommodated state, asin FIG. 5( b), except that here the apparent effect of viewing a“distant object” is achieved by moving a lens 116 from a close distance(c) to a farther distance (c+e) from the cornea. A retinal image 136formed in this way has the same size as image 134 of FIG. 5( b).Positional translation of lens 116 or image surface 120 as shown in FIG.5( d) may be effected by means of a positional actuator 138 controlledin a position control system such as is shown in FIG. 6. A positionsensor 140 may be provided to provide a feedback signal on a line 142 tothe control 44 of FIG. 3. Proportional, proportional-plus-integral,proportional-plus-integral-plus-derivative, any combination ofproportional, integral or derivative, or any other well known closedloop control system techniques may be used in the control 44 to maintainthe lens 114, image surface 120 or means 98 at the commanded position.

It should be understood that the approach selected for the variablemagnification means 98 of FIG. 3 may be taken from a wide variety ofpossible approaches. These include liquid lenses with refractiveproperties that change according to changes in the volume of liquidbetween one or more movable or flexible surfaces of such lenses. See,for example, U.S. Pat. No. 4,289,379 to Michelet which discloses aprinciple which may utilized for a variable magnification lens.Therefore, the specific approaches described herein should be understoodas being merely incidental choices and not in any way limiting thecentral core idea, as expressed in several of the dependent claims, ofpresenting the mixed images to a passive viewer at various apparentdistances. Thus, for example, as previously suggested, the position ofthe image plane 118 could also be changed in a manner similar to thatshown in FIG. 5( d). There, the image plane is movable in a directionfurther away from the eye 42 as shown an added distance f for a totaldistance of d+f, and is also movable in the opposite direction (notshown) closer than distance d to the eye 42. The object is to achievethe same effect, i.e., of presenting images at various apparentdistances.

FIG. 7( a) is a simplified illustration of a portion of a displaysurface 144 on, for example, a cathode ray tube (CRT) display scanned ina manner consistent with the object of the present invention. Although aCRT display is described, it will be understood that other types ofdisplays may be used equivalently. These include but are not limited tothe various types of electroluminescent (EL), gas discharge, liquidcrystal (LC), ferroelectric, electrochromic, electrophoretic, vacuumfluorescent, thin-film transistor (TFT), silicon switches, semiconductorswitches with capacitors, metal-insulator-metal (MIM), combinations ofthe foregoing, or any of the other emerging display technologiespresently being developed or to be developed which are reasonably withinthe scope of the claims of this patent during the term hereof.

The scanning method described is consistent with a scanning methoddescribed in U.S. Pat. No. 4,513,317 but the scanning method describedin that patent and those described in that patent and those described inthis patent are in no way limiting to the present invention, i.e., as tothe method of constructing a fine detail image area amidst a coarseimage detail area. For the sake of shortening this specification, thevarious Figures and related textual material of that patent will beomitted, but are hereby expressly incorporated by reference.

Five coarse scanning lines 146, 148, 150, 152, 154 are illustrated inFIG. 7( a) herein, in truncated form. These sections of lines representonly a small portion of the total number of coarse scanning lines on theimage pickup or display surface 144, the total number being up to thedesigner, but typically, without limitation, being on the order ofhundreds. For example, eighty, a hundred, two, five, twenty or anynumber hundred or fraction of a hundred coarse scanning lines might beselected to scan a display surface. These might be assembled adjacentlyin sequence or may be interlaced in some convenient manner to eliminateflicker by providing more numerous images, albeit of less detail. Acoarsely focused electron beam is one means of producing a low detailarea 315 such as is shown in FIG. 2 of U.S. Pat. No. 4,513,317.

Also pictured in FIG. 7( a) herein are four fine scanning lines withineach of the three coarse scanning lines 148, 150, 152 which together (astwelve fine lines) very roughly correspond to the fine detail area 310of FIG. 2 of U.S. Pat. No. 4,513,317. These can be formed byinterrupting the coarse scanning beam which forms coarse left-to-rightscanning line 148 at a vertical boundary 156 and a horizontal boundary157, at the intersection of boundary lines 156, 157, sequentiallycommencing a fine four line scan 158, 160, 162, 164 within coarse line148, each line of which terminates at a vertical boundary 166 and, in asimilar manner, within each of the coarse lines 150, 152, respectively,forming fine lines 168, 170, 172, 174 and 176, 178, 180, 182, stoppingthe fine lines at a line 183. Together, the twelve fine scan lines ofFIG. 7( a) form a small, highly detailed image area amidst a generallycoarsely detailed CRT display surface. Of course, the coarsely andfinely scanned portions may be scanned during separate periods of afield or a frame or even in different frames. This is a matter ofchoice.

The position of the highly detailed area on the display surface iscaused to change according, for example, to the magnitude of the eyeposition signal 80 of FIG. 2, or the magnitude of one or both of the eyeposition signals 110, 112 of FIG. 4. This can be carried out using muchof the camera apparatus illustrated in FIG. 5 of U.S. Pat. No. 4,513,317but modified to include monitoring a cameraman's eye or eyes to controlthe fine detail scanning at the camera end rather than an active remoteviewer at the display end. The highly detailed image area is replicatedin the image space 22 by means of the display apparatus 26 which couldbe carried out as well using the display apparatus of FIG. 6 of U.S.Pat. No. 4,513,317 modified to omit monitoring the remote viewer's eye.

The modified composite video signal illustrated for one frame's field inFIG. 4 of U.S. Pat. No. 4,513,317 is an acceptable format for encodingthe coarse and fine image information as well as the positioninformation in video signal form but should in no way be considered alimitation on the present claims since there are numerous possible otherways of encoding the coarse and fine image information. For example, themethods described in U.S. Pat. No. 3,491,200 could, without limitation,be used as well. The apparatus and methods suggested in U.S. Pat. No.4,513,317 have been cited for illustrative purposes only.

It will be understood that although the technologies used forillustrative purposes herein are of the black and white type, theinventive concepts disclosed and claimed herein are of course notrestricted thereto since the same concepts may be utilized in thecorresponding color type.

FIG. 7( b) is an illustration of a display surface on, for example, aflat panel display having a large number of controllable cells orpixels, each capable, e.g., of displaying various shades of gray or,alternatively, of being used as binary cells, each used for displayingone shade of white and one of black only but set up for scanning bymeans of pulse width modulation for integration in the viewer's visualapparatus over time as various shades of gray. (The achievement of goodgray scale performance requires a contrast ratio of about 50 to 1, withthe brightness/contrast either continuously variable or, if quantized,controllable into at least sixteen logarithmically spaced steps.Sixty-four shades are desired for good, aesthetically pleasing picturequality). Thus, it is known in the art to use both an analog signalamplitude and a digital pulse width modulated signal for controllingpixel luminance.

Although analog techniques of achieving gray scale are generallyvisually superior and easier to implement, many types of flat paneldisplays cannot use analog control for a variety of reasons. Besides thepulse-width modulation technique described above, other techniques suchas multiple display cells per pixel, changing the element's duty factoron a frame-time basis, introducing panel instabilities, and others areknown.

The fundamental parameter, in any case, is the percentage of timeavailable to address a pixel which directly limits the averagebrightness of a display panel. Luminance or contrast is affected by theamount of instantaneous power that a display material, and othercomponents, can tolerate during the duty cycle. Some display materials(LCDs) do respond to short electric pulses and some (electroluminescentand cathodoluminescent materials) become more efficient when high-powerpulses are applied.

In one type of multiplexing, each pixel is turned on for a fraction ofthe time equal to the frame time divided by the number of pixels. If theframe is repeated at a frequency above the critical flicker frequency,the viewer's visual apparatus will integrate the pulses and flicker willnot be seen. For a frame repetition rate comparable to commercial TV,this scheme only allows 140 ns to address a pixel. In another type ofmultiplexing, the time “on” per pixel can be increased to approximately65 microseconds by line-at-a-time addressing, i.e., the dwell time perpixel is increased by a multiple equal to the number of columns. Thus,the duty cycle per pixel is equal to the frame time divided by thenumber of rows. All the columns can thus be randomly addressed inparallel without further complicating cross-coupling in any other row.

The basic principles presented are the same for a color display, but aredescribed here in terms of black and white because the principles arethe same because the colors presented by a color display will have alightness or luminance attribute which is directly analogous to a grayscale. Therefore, it will be understood that the scope of the inventioncovers the presentation of color images, as well as black and white.Moreover, the scope of the invention is not limited to presenting imagesby means of a liquid crystal, electroluminescent, CRT, gas plasma,electrophoretic, electrochromic or any other specific display type.Other suitable display technologies, both analog and digital, emissiveand nonemissive, are certainly within the scope of the invention. Ofcourse, the choice of display technology will determine the character ofthe individual picture elements (pixels) and the manner of assemblingcoarse and fine composite images. For example, a CRT approach mightutilize a coarsely focused electron beam for scanning wide image areasand a finely focused electron beam for scanning the small, highlydetailed portion of the composite image. Such a CRT scanning approachfor a retinally stabilized differential resolution television display isdisclosed by Ruoff, Jr. in U.S. Pat. No. 4,513,317, as described abovein connection with FIG. 7( a). For flat panel displays the pixels may beaddressed in a number of different ways including but not limited todirect (such as is used in alphanumeric and very large displays),scanning (used in CRTs and laser displays), parallel (used in projectiondisplays and large screens), matrix (generally used in flat paneldisplays), and grid (used in flat panel displays and flat CRTs). Flatpanel displays generally use only the matrix or grid techniques.

Although flat panel embodiments of the present invention are not limitedto a matrix of pixels arranged orthogonally, that arrangement is ingeneral use and will be used for descriptive purposes. Each pixel has arow and column address. A matrix 480 by 500 columns has been selectedfor the descriptive purposes of this specification. Such is comparableto commercial TV, having a total of 240,000 addressable pixels. To usean individual wire for each pixel is virtually impossible unless thematrix is the size of a billboard. However, individual conductors can beused for each row and each column.

A pixel was formerly usually a two terminal device but three terminaldevices are becoming much more common. When a voltage is applied betweenthe two terminals of a two terminal device, a contrast phenomenonoccurs. In an orthogonal matrix addressed display, each pixel has oneterminal ganged together with other pixels in a row and the secondterminal ganged together with the other pixels in an orthogonal column.When a pixel is selected, its row and column leads are energized. Noneof the other pixels are selected intentionally, but “sneak paths” doexist. A fraction of the applied voltage exists across all pixels. Ifthe pixel image medium is approximately linear, then a correspondingimage background occurs and destroys the image contrast. Analogoushookups are used for stimulating similar contrast phenomena in threeterminal devices.

The problem of addressing a matrix is compounded further when a secondpixel is selected simultaneously on another row and column. Ideally,only the two pixels would display. In reality, however, two additionalpixels would also see the full voltage across their terminals. The fourpixels would constitute the corners of a rectangle formed by theintersection of the two column lines and the two column lines and tworow lines.

The sneak-paths and four-pixels problem are fundamental to thematrix-addressing technique. The sneak-paths problem is solved by usingpixel devices with nonlinear properties, i.e., devices that are notexcited at fractional voltages. The four-pixels problem is solved byusing a technique called line-at-a-time addressing, where only onehorizontal line lead is addressed at any instant. All column leads areaddressed in parallel. This time sharing, or multiplexing, greatlyreduces the time allowed for pixel addressing. A display material mustbe responsive to a short duty cycle if it is to be multiplexed. Althoughthere are few materials that possess these two properties, thenonlinearity requirement can be achieved by adding a switch at eachpixel matrix junction in the form of a diode, a field-effect transistor(FET), or a thin film transistor (TFT); the short duty cycle and lack offast display response can be overcome by adding a capacitor at eachjunction. It is charged at multiplexing rates and discharges to excitethe display material during the balance of the frame time.

A portion of a display surface is illustrated in FIG. 7( b) for oneparticular composite image design. Large squares 184 are each made up ofsixteen small pixels 186. All the large squares 184 are controlled atthe same shade of gray and are assembled in coarse horizontal scanninglines to form lesser detailed image areas. Each coarse line is foursmall pixels wide. A small section of seven such coarse horizontalscanning lines is shown in FIG. 7( b). Such coarse squares may beassembled in any suitable manner, depending on the particular displaytechnology employed and depending upon convenience. One large square iscentered on the intersection 188 of lines 190 and 192 and is made up ofsixteen pixels, each individually controllable for display at variousshades of gray. Twelve intermediately sized squares 194 of four pixelseach, are assembled around the periphery of the central square centeredat 188 to help form a highly detailed image area. The intermediatelysized squares 194 are located around the periphery of the single,fine-detail large square centered at 188. All of the four pixels in anygiven intermediately sized square 194 are controlled for display at thesame shade of gray. Again, a fine-detail large square of fine pixelssuch as that illustrated centered at 188 and the plurality ofintermediately detailed squares 194 may be assembled in any convenientmanner. The use of intermediately sized squares smoothes the transitionfrom high detail to low detail image area in a manner very roughlysimilar to the population fall off of cones from the center of thefovea. See, for example, FIG. 2 of U.S. Pat. No. 3,953,111 and FIGS. 3and 9 of U.S. Pat. No. 4,479,784 for plots of visual acuity as afunction of angular displacement from the fovea. Of course, a morecomplicated structure would be necessary to truly replicate the fovealfall off and such is certainly within the scope of the presently claimedinvention.

The compositeness of the assembled image may also be effected by anyconvenient method. For example, each successive composite image may becomposed by firstly assembling and presently a coarsely detailed imageover the entire field of view (e.g., in a first field or frame) andsecondly assembly and presenting a finely detailed image over only asmall portion of the full field of view (in a second field or frame).Other, equally satisfactory approaches may of course be employed. Foranother example, the coarsely and finely detailed portions may beassembled at the same time (in the same frame or even the same field).The word “composite” is appropriate in some circumstances, according tothe present invention, to describe the melded image assembled as, forexample, in FIG. 7( b). However, for certain other methods of assemblingpicture elements or pixels to achieve the same object, the word“composite” is not quite right. For example, FIGS. 8-12, withoutlimitation, show several other approaches which are possible forconstructing images having both highly detailed and lesser detailedcomponents. Most of these would most likely be implemented using a CRTapproach in which a scanning electron beam is used to trace the lineswhich form the scanning patterns illustrated. There are no easilydiscernible boundaries between the highly and lesser detailed areas inFIGS. 8-12, at least, and thus there is no “composite” image, as in FIG.7( b), where image components of clearly distinguishable resolution arebrought together. A broader term, which covers both the compositeapproach of FIG. 7( b) and the approaches of FIGS. 8-12 is “mixed”image. This term is used herein to cover both approaches.

FIGS. 8-12 show various scanning patterns which represent only some ofthe many patterns which might be employed in achieving the objects ofthe present invention. FIG. 8 shows a spiral scanning pattern in whichthe highly detailed image information is concentrated in the center ofthe spiral. The spiral may be more tightly wound in the center thanwould be called for by a strict application of an Archemedian spiral,for example, in order to intensify the high detail effect at the center.The entire scanning pattern is mobile as a unit. FIG. 9 shows a patternmade up of a plurality of intersecting lines. FIG. 10 shows a patternmade up of a plurality of concentric circles. The circles in the centermay be more closely spaced than those at the periphery. FIG. 11 shows apinwheel pattern. FIG. 12 shows a coarse raster pattern for backgroundand a fine raster pattern for detailed image information. The small,detailed raster may be made mobile with respect to a stationarybackground (stationary, that is, in the absence of head movements), asimplied by FIG. 12 and as described in detail above in connection withFIGS. 7( a) and 7(b), or may be moved in tandem with the background.

Other scanning techniques not shown might include variations onLissajous scanning, decreasing rectangle, perpendicular, angular, sinewave, etc., in which a small, highly detailed area is inherent or isintroduced. In all cases, the human visual process is simulated bychanging the portion of the object space imaged by the finely detailedimage area. This may be accomplished by either of two methods. The firstis to make the finely detailed area mobile with respect to the lesserdetailed area to simulate ductions. Using this approach, changes in theimage content of the lesser detailed area is solely indicative of headmovement. The second way is to change both areas at the same time tosimulate, at any given moment, either ductions solely or head movementssolely, or both ductions and head movements at the same time. Forexample, supposing that in FIG. 7( b), the full image area (only afragment of which is shown) covers the full field of view of a left eyein which the upper portion of the image is bounded by the left brow, theright by the nose, the bottom by the left cheek and the left by thelimits of peripheral vision. If the first method is chosen then theimage content of the large squares 184 will remain the same for as longas the simulated eye's head remains stationary. For a case in which theimages are formed by means of a cameraman, this corresponds to for aslong as the cameraman's head remains stationary. The image content ofthe highly detailed component 188, 194, on the other hand, changes tosimulate eye rotations. It changes its image content by changing itsposition within the boundaries described. For the example, this mightmean that the highly detailed component in FIG. 7( b) would change itsposition from the upper left center portion of FIG. 7( b) to the lowerright center portion. This would of course induce a change in thedirection of the visual axis of the viewer in the image space, forexample, from an elevation away from the nose to a depression toward thenose by means of a combined infraduction and adduction. This changewould correspond to a similar change in the direction of the simulatedeye's visual axis in the object space.

On the other hand, if the second method is chosen, the highly and lesserdetailed areas will move in tandem to simulate eye rotations while atthe same time both the lesser and highly detailed areas will changetheir image content to simulate head movement, even if the simulatedeye's axis remains fixed relative to the head.

FIG. 13( a) is a simplified block diagram illustration of an apparatus196, similar to the control 44 of FIG. 2, according to the presentinvention, without limitation. The apparatus of FIG. 13( a) isparticularly suited for a display such as is used to produce a mosaicversion of FIG. 12, for a fine raster either fixed or mobile withrespect to the coarse raster and, which is also amenable to beingconstructed using a camera pickup 66 such as is illustrated in FIG. 2using a light sensitive matrix such as is illustrated in FIG. 13( c) anddisplayed using a matrix display such as is illustrated in FIG. 13( b).A video amplifier 198 is responsive, for example and not by way oflimitation, to a composite video signal on the line 46. The videoamplifier provides an increased amplitude version of the signal on theline 46 on a line 200 to a synchronizing signal separator or stripper202.

A control unit 204 is also responsive to the amplified video signal online 200 and includes control circuitry for supplying a coarse imageblanking signal on a line 206, a coarse image information signal on aline 208, a fine image address signal on a line 210, a fine imagesteering signal on a line 212 and a fine image signal on a line 214.

A display structure such as, without limitation, a matrix display 216,is shown in FIG. 13( b) and is responsive to control signals from thecontrol 196 of FIG. 13( a) for displaying pixels having the properlightness or luminance which, when displayed together, effect thedesired image pattern. The sync separator 202 of FIG. 13( a) stripshorizontal synchronizing pulses from an amplified composite video signalon line 200 and provides them on a line 218 to a horizontalsynchronizing circuit 220 which provides a fine row sync pulse on a line222 to the display of FIG. 13( b). The horizontal sync circuit alsoprovides a coarse row sync pulse on line 224. The sync separator 202also strips off a vertical synchronizing pulse during the verticalsynchronizing period between each coarse field and provides a signalindicative thereof on a line 242 to a vertical synchronizing andblanking circuit 228 which provides a fine field sync signal on a line230 to the display. Similarly, a coarse field sync signal is provided ona line 232. The control within the display will control the sequentialdisplay of the proper rows (horizontal lines) and fields according tothe horizontal and vertical sync signals, respectively.

Referring back to FIG. 13( b), a 500×400 matrix display 234 is shown,without limitation, for providing mixed images in response to the imageand control signals of FIG. 13( a). The illustrated matrix is “scanned”one line at a time for fine imaging using an eighty line addresser 236which addresses any eighty adjacent lines out of the four hundredavailable in sequence starting at a line addressed by the horizontaladdress signal line 210 and continuing in sequence until all eightylines are scanned. Prior to each such line scan, one hundred fine imagesignals are assembled in a charge transfer device 238 in response to theimage signal on the line 214 and are demultiplexed or otherwise steeredinto one hundred adjacent vertical lines by a 100×500 steering circuit240 in response to the vertical address signal on the line 212. Theeighty line scan of one hundred pixels per line produces a fine imagerectangular area of eight thousand pixels out of the two hundredthousand pixels in the matrix, representing 4% of the surface area. Therelative position of the fine image area will of course change fromsuccessive image to successive image.

The steering circuit 240 may, for example, comprise a first plurality ofdemultiplexers in a first stage and a second selected plurality ofdemultiplexers in a second stage. For example, each of the one hundredsignals from the charge transfer device 238 may be connected at a firststage as an input to a corresponding 1-to-20 demultiplexer. Each of thetwenty outputs of each of the one hundred first stage demultiplexers maybe connected as an input to a 1-to-25 demultiplexer to form a secondstage. This would allow any of the one hundred outputs of the device 238to be switched to any of the five hundred columns of the matrix. And ofcourse a selected group of a hundred adacent columns may have allhundred outputs switched to them. Since this involves a very largenumber of connections, such may be reduced by various techniques. Onetechnique would be to only allow the first of one hundred fine lines tobe positioned within the range of one to four hundred of the fivehundred lines. In that case, the one hundredth fine scanning lines wouldbe confined to lines one hundred to five hundred. Other techniques,without limitation would include reducing the size of the highlydetailed scanning area, by skipping scanning lines or other similartechniques. Any other switching scheme may be used as well.

The coarse part of the mixed image may be provided by “scanning” fiveadjacent lines at a time, each five-line group being treated as a singleline of one hundred (5 pixel by 5 pixel) coarse picture elements. Each5×5 pixel coarse picture element may be treated as an elementary pixel.I.e., all twenty-five pixels assume the same shade of gray (for blackand white applications). The matrix may thus be coarsely scanned fromtop to bottom by eighty coarse horizontal scanning lines of one hundredcoarse picture elements each. This may be accomplished using anothereighty line addresser 242 which provides output signals sequentially oneighty separate output lines 244, 246 (not shown), . . . , 248 (notshown), 250 (not shown), 252 (not shown), . . . , 254 (not shown), 258(not shown), . . . , 260 (not shown), 262 (not shown), 264 inconjunction with eighty separate splitters 266 ₁, 266 ₂ . . . , 266 ₇₉,266 ₈₀, each being associated with one of the eighty output signals fromthe addresser 242. Each splitter splits each output signal it receivesinto five similar signals for addressing five separate horizontal linesat a time. Thus, all four hundred lines of the matrix may be coarselyscanned using eighty coarse scanning signals in conjunction with eightysplitters. During the period in which a given coarse line is scanned, aone hundred cell charge transfer device 230, which may be thought of asan analog shift register, provides 132 stored signals it had previouslyreceived serially from coarse image signal line 208 on lines 270 ₁, 270₂, . . . , 270 ₉₉, 270 ₁₀₀ (100 signal lines) to a corresponding gate272 ₁, 272 ₂, . . . , 272 ₉₉, 272 ₁₀₀ (100 gates), each gate beingresponsive to the coarse scanning gate signal on the line 206. Thissignal serves the purpose of blocking the display of image informationover portions of selected horizontal scanning lines that are scannedinstead by fine scanning lines. Thus, in the illustration, the actualcoarse image intelligence is blocked from being displayed. For each fullframe, there will be twenty adjacent gates selected out of the onehundred gates 272 ₁, . . . , 272 ₁₀₀ in sequence which will be used forblocking coarse image information. The particular timing and position oftwenty adjacent blocking gates will of course change from frame to framedepending on the magnitude of the eye position signal 80 (FIG. 2), whichis translated by control unit 204 or 82 into the proper timing for thesignal on the line 206 and also into the proper control signals forsignal lines 208, 210, 212, 214, also by either the control 82 of FIG. 2or the control 204 of FIG. 13( a). Thus, during sixteen of the eightycoarse scanning lines, only eighty of the one hundred signals on lines270 ₁ through 270 ₁₀₀ will get through to a corresponding eighty out ofa hundred splitters 278 ₁, 278 ₂, . . . , 278 ₉₉, 278 ₁₀₀. There willthus be a rectangular “hole” of twenty consecutive vertical linessomewhere on the screen, at any given moment, according to the directionof the eye of the cameraman.

Of course, a corresponding scanning process will have been mostadvantageously, but not necessarily (e.g., alternately via computersimulation or animation), carried out in exactly the same manner in thecamera apparatus 58 of FIG. 2 except in reverse. In that case, the sync,addressing and control signals on lines 210, 206, 208, 214 are firstgenerated in the control 82 in the image space for controlling thescanning of the video camera and are encoded into the composite videosignal on line 40 provided for display purposes and the control 204 ofFIG. 13( a) will in that case have little or no computational burdenimposed on it. On the other hand, it is conceivable that another designcould have uniform scanning at the camera end which is only transformedinto mixed images at the display end with the aid of the magnitude ofthe eye position signal encoded in a single composite video signal online 40 or a plurality of equivalent component video signals. Of course,this would be bandwidth wasteful, but nonetheless one of many differentpossible solutions to the encodement problem. In addition, if the samebasic approach as used in the display of FIG. 13( b) is taken in reversein a camera, the splitters 278 ₁, 278 ₂, . . . , 278 ₉₉, 278 ₁₀₀ wouldhave to serve an averaging function for ensuring that each 5×5 coarsepicture element is of uniform contrast or luminance. This process ispreferably carried out at the camera end but could be done at thedisplay end.

For example, a typical field is scanned first by a coarse field andthen, if so designed, by a fine field. Of course, an entire coarse framemay be scanned first, followed by an entire fine frame. By way ofexample only, a frame is first coarsely scanned, in a coarse field, fivelines at a time over the entire surface of the matrix with large (5×5)pixel picture elements except for an 8,000 elemental pixel (100×80pixels) rectangle which is not scanned coarsely since it is to blockedat the proper position by the signal on line 206. For the schemedescribed, this means sixteen successive coarse scanning lines areblanked out for twenty successive vertical groups of five coarse pixelseach. The frame may then be finely scanned in the next field over thesmall (100×80) rectangular area blanked out in the previous coarsefield. If each of the pixels has an individual capacitor associated withit, this means that the capacitors associated with the coarsely scannedarea will have the same charging time as those of the finely scannedarea and no special voltage adjustments need be made for one fieldgetting less charging time than the other. This simplifies the matchingof the brightness of the fine field to that of the coarse field.Although not described in detail, but as will be understood by thoseskilled in the art of television engineering, interlacing may be used toeliminate flicker.

A portion of a solid state video camera such as the camera 66 of FIG. 2is shown without limitation in FIG. 13( c). There, a matrix 286 whichmay be a CCD, MOST, or other type solid state imaging matrix is scannedcoarsely and then finely, as in the display 234 described above in FIG.13( b), except in the reverse process, by gathering intelligence ratherthan providing it, in order to produce the image signals on line 86 ofFIG. 2, which are ultimately conditioned, encoded, transmitted, decodedand translated into the signals appearing, respectively, on lines 208,214 of FIG. 13( b). As suggested, unlike the display matrix 234 of FIG.13( b), the camera matrix 286 of FIG. 13( c) has image informationsignals flowing from the matrix to the control electronics, i.e., thereverse process. The control elements are similar, playing the sameroles except in reverse. The only major exception is one hundred coarseaveragers 288 ₁, 288 ₂, . . . , 288 ₉₉, 288 ₁₀₀, which are eachresponsive to image information from twenty-five pixels, each pixelcorresponding to one of the twenty-five intersections of five adjacentrows with five adjacent columns in a coarse picture element. Theaveragers serve the function of averaging the total charge received intoan average value for the intersecting five rows and five columns. Thus,the five addressed rows (for each coarsely scanned line) contribute onepixel each to each of the five columns for a total of twenty-five pixelsto be averaged in each averager for each coarsely scanned oversizedpicture element. It is this averaged magnitude which is provided on onehundred signal lines 292 ₁, 292 ₂, . . . , 292 ₉₉, 292 ₁₀₀ to blockinggates 296 ₁, . . . , 296 ₁₀₀. The blocking gates 296 ₁, . . . , 296 ₁₀₀will not be needed if blocking gates 272 ₁, . . . , 272 ₁₀₀ are used inthe display of FIG. 13( b). Similarly, if the blocking gates of FIG. 13(c) are used then the blocking gates 272 ₁, . . . , 272 ₁₀₀ of FIG. 13(b) need not be present. This latter alternative is of course usuallypreferred for economic reasons due to a large number of displays asopposed to a small number of cameras. It will thus be understood thattwenty averagers (a hundred pixels or twenty large picture elements (5×5pixels) must be blocked out from sixteen coarse horizontal scanninglines during each coarse scanning field. The eighty averaged signalswhich do get through the blocking gates plus twenty black level signalsfrom the twenty selected gates will be transmitted to a one hundred cellcharge transfer device 300 over one hundred signal lines 302, . . . ,304. Thus, during sixteen out of the eighty coarse scanning line periodsduring each coarse field, there will be twenty adjacent black levelsignals stored somewhere within the 100 cell charge transfer device.These will later be translated into eighty short lines (in groups offive lines) of a hundred “black” level pixels somewhere on the displayfor fill-in during the fine scanning field. The device 300 provides acoarse image signal on a line 306 which will eventually be representedby the signal 208 in FIG. 13( b); for the moment, however, it isprovided over signal line 86 of FIG. 2 to the control 82 for encodementinto a composite or component video signal provided on line 88 fortransmission to the image space 22. Referring back to FIG. 13( c), itwill be understood that the eighty coarse scanning lines are addressedone at a time (actually five rows of pixels at a time) by an eighty lineaddresser 308 as triggered by coarse row sync pulses on line 310provided over line 84 by control 82. There are eighty splitters 312, . .. , 314 which are used to address eighty corresponding groups of fiverows. The fine field of each frame is addressed by an eighty lineaddresser 316 which simply energizes one row at a time until eightysuccessive lines have been energized. The addresser 316 is triggered bya control signal on a line 318 which commands the addresser to startaddressing at a particular line out of the four hundred available.Control signal line 318 may also be provided over the signal line 84 ofFIG. 2 along with control signals 320, 318, 322. A five hundred by onehundred selection circuit 324 is connected to all five hundred verticalcolumns but only selects one hundred columns for each of the eightyindividually addressed fine lines and provides each 100-pixel line insuccession to a one hundred cell charge transfer device 326, as steeredfrom the matrix 286 through the selection circuit 324 by the controlsignal on line 322. The selection circuit 324 may be similar to thesteering circuit 240 of FIG. 13( b) except using multiplexers instead ofdemultiplexers. A fine field image signal is provided on a line 328which will eventually become translated into the signal on line 214 ofFIG. 13( b).

The format for the video signal on line 40 may, without limitation,assume a format particularly adapted for use in any of the embodimentsof the present invention such as disclosed in FIGS. 2, 12 & 13(a) andthus may be selected from a wide variety of alternative formats known inthe video art. The format may, for example only, and not by way oflimitation, be constructed in a manner similar to that shown in FIGS. 3& 4 of U.S. Pat. No. 4,513,317 to Ruoff for use in the apparatus 26 ofFIG. 2 herein for presenting images constructed in the manner of FIGS.7( a) & 12. The same approach can be used for the embodiment of FIGS.13( a), 13(b) and 13(c) except for the modification, without limitation,that the relative time allotted within each field for scanning the highdetail area will be equal to that allotted for scanning the low detailarea. For the case where the images are presented at various apparentdistances, an additional signal is necessary in the period designated T1of Ruoff for the distance information. This may be accommodated bysimply allotting an extra horizontal “line” from period T1 and insertingan analog signal level similar in concept to the Vx and Vy levels shownin FIG. 4 of U.S. Pat. No. 4,513,317. For the stereoscopic embodiment ofFIG. 16 herein, alternate fields may be allocated to alternatesuccessive images for alternate presentation to the left and right eyesof the passive viewer. For the multiphonic embodiment disclosed inconnection with FIG. 17 below, the audio portion of the typicalcomposite video signal is suitably modified (see, e.g., Part VI ofTelevision Technology Today, Ed. by T. S. Rzeszewski, IEEE Press, 1985)to account for the multiplicity of speakers so that the proper speakerreceives the proper signal at the proper time.

Referring now to FIG. 14, a composite video signal waveform 330 is shownwithout limitation, such as may be formed, again without limitation, inthe apparatus 58 of FIG. 2 and used to produce, for example, the displayof FIG. 7( b) and as might appear on line 40 of FIG. 2. The video signalmay be structurally equivalent to that of commercial television in orderto take advantage of at least part of the existing infrastructure, i.e.,the broadest network and also videocassette recorders (VCRs). VCRcassettes offer a convenient storage facility for storing videoproductions made in accordance with the present invention, although theexisting cameras and TV receivers are not readily compatible, due to thedifferent scanning requirements. It is, however, not inconceivable thatthe present invention could be used on conventional TV receivers. Thiscould be accomplished, for example, by using the present level ofcommercial television resolution for the high detail portion of theimage only and deliberately degrading the remaining portion for the lowdetail resolution areas. Of course, the composite video signal used toimplement the present invention need not bear any relation whatever tothe structure of the composite video signal presently used in commercialtelevision. That is purely a matter of expedience. And, of course,separate component video signals may be used in lieu of a compositevideo signal.

The conventional odd 332 and even 334 fields of a single conventionalframe are shown. Each field, however, is broken down into four subfields336, 338, 340, 342 and 344, 346, 348, 350, respectively, each group offour being labeled T1, T2, T3, T4 in the Figure. The subfields may be ofequal periods or may be different, depending on the design. A verticalblanking period 352 separates each field. The first subfield (T1) ineach field is used to coarsely “scan” the odd lines from the top of thedisplay to the bottom. Referring back to FIG. 7( b), this wouldcorrespond to scanning alternate groups 354, 356, 358 of fourarbitrarily designated “odd” lines at a time. Thus, it will beappreciated that the first coarse “odd” line 354 will be made up of theconventional lines 1-4, the second coarse odd line 356 of conventionallines 9-12, the third coarse odd line 358 of conventional lines 17-20,and so on. Thus, if one were to split the standard odd field of 262.5lines into groups with four adjacent lines in each group, there are nowonly 65.625 coarse scanning lines in the first quarter of the odd field.Blanking pulses (not shown) of appropriate duration are inserted betweenselected horizontal sync pulses in order to blank out coarse scanning inthe area of the display where fine scanning is to take place. Forexample, coarse line 354 is blanked out in the rectangular area coveredby lines 360, 362 and also eight of the sixteen pixels for the areacentered at 188. The rest of the fine detail area is blanked out in thesame relative position between the two horizontal sync pulses markingthe beginning and end of the first even coarse scanning line 364. Thesecond subfield (T2) in each field is used to scan the intermediatelyfine areas by means of four moderately fine short scanning lines 360,362, 366, 368. Since only a very small portion of the total screen areawill be utilized for image information during each subframe T2, it willbe appreciated that the short line pairs 360, 362, 366, 368 may bescanned more than once, in fact a large number of times during T2,depending, in the various possible designs, on the number of groups 194selected to simulate the macula area. In such a case, the relativebrightness of the high detail area will be too great in comparison tothat of the low detail area unless the intensity of each high detailpixel is adjusted to a lower intensity level such that the cumulative orintegrated intensity over a field, frame or similar interval is the sameas for the low detail area. Preferably, the relative duration of T2 maybe reduced in comparison to the coarse scanning period so as to permittime for only a single scan. The number of groups and the number oftimes selected for repetitive scanning is of course a design choice, asis the format of FIG. 7( b) itself. The third subfield (T3) in eachfield is used to coarsely scan the even lines. This means that coarseeven lines 364, 370, etc., corresponding to conventional lines 5-8,13-16, etc., will be scanned during T3. The fourth subfield (T4) in eachfield is used to individually scan the lines of pixels which make up thecentral area centered at 188. Since in using this method there are onlyfour lines of four pixels each, these lines may be scanned repeatedlyduring T4. Since there will be a great many (approximately sixty-six)horizontal sync pulses available, it would be possible to repeatedlyscan the pattern a very large number of times. Again, the relativebrightness will be too great unless the intensity of the pixels in eacharea is adjusted in inverse proportion to the number of times scanned ina frame, field or similar period. If the high intensity area is scannedonly once, the relative duration of T4 should be reduced accordingly,the scanning speed should be proportionately slowed down with anaccompanying current reduction, or the current should be increased toincrease the brightness.

It will be understood that the pattern of FIG. 7( b) has beenconstructed during only one field of a conventional composite videosignal, using four separate and interleaved subfields as buildingblocks. The second conventional field 334 of FIG. 14 is used toconstruct an entire frame in exactly the same way so that the framerepetition rate is effectively doubled. This is important for removingthe flicker effect for low brightness displays and may also be used forthe stereoscopic embodiment to be described below where the odd fieldperiod is used for left eye images and the even field for right eyeimages.

It should be understood that a composite signal suitable for use in thedisplay of FIG. 13( b) may be constructed, without limitation, using asimilar design strategy except omitting intermediately detailedscanning.

It should also be understood that although not shown in FIG. 14, certain“lines” must be utilized for storing nonimage information such as isshown in period t1 of FIG. 4 of U.S. Pat. No. 4,513,317. These lines maycontain not only Vx and Vy information for locating the high detail areabut also distance information for magnifying and shifting the image.

Referring now to FIG. 15, a very simple method and means forconstructing (at the camera end 372) and presenting (at the display end374) panoramic mixed images is shown for purposes only of illustratingone way of presenting such images, and certainly should not beconsidered as limiting the scope of the invention. A lens 376, such aspherical planoconvex lens in a camera, such as the camera 66 of FIG. 2,is responsive to input light signals such as on line 32 for providingcamera image signals on a typical line 378 to a light sensitive surface380 in the camera. The light sensitive surface provides an encoded imagesignal on the line 382 to a storage medium 384 such as a broadcastnetwork near simultaneous pickup and display or a VCR for recordal andlater playback for providing an output signal on a line 386 to a displaysurface 388 in a video display which forms optical image signals on atypical line 390 to another lens 392 such as a spherical planoconvexlens which provides a typical output optical image signal on a line 394for a passive viewer's eye. Of course, as mentioned, the encoded imagesignal need not be stored by a VCR but may be stored on a video disc, beprovided directly, or broadcast as in FIG. 2, among other methods. Foran embodiment in which accommodation is induced in the passive viewer, avariable magnification means, such as a laterally movable biconvex lensmay be interposed between the display 388 and the planoconvex lens 392.Or, the display surface 388 may be movable. Of course, other moresophisticated means of providing the desired panoramic effect are alsocontemplated and are within the scope of the present invention. Onlyone, very simple approach has been illustrated. For color embodiments,for example, the corrected lenses disclosed in U.S. Pat. No. 4,406,532may be used. Although this simple method and means is a highlyadvantageous approach to the problem of presenting panoramic mixedimages, it is by no means the only approach possible and shouldtherefore in no way be considered limiting. For instance, thehemispherical images of U.S. Pat. No. 2,719,457 may in principle beprovided.

Referring now to FIG. 16, a stereoscopic camera embodiment and twostereoscopic display embodiments of the present invention are thereillustrated. An object space 400 contains a cameraman 402 having a righteye 404 and a left eye 406 which are respectively monitored for positionby eye position monitors 408, 410 via signal lines 412, 414, each ofwhich may represent more than one signal, e.g., both an infraredillumination beam and an infrared reflection beam, the reflection beamhaving a magnitude indicative of the monitored eye's angular position.Each eye position monitor provides at least one eye position signal 416,418 to respective control units 420, 422, which in turn provide scanningcontrol signal lines 424, 426 to respective video cameras 428, 430. Theeye position monitors may measure one or more axes of rotation of theeye, i.e., both horizontal and vertical ductions and torsions as well.If torsions are not measured they can be computed or looked up, ifdesired, based on the point of convergence and torsions that can bepredicted according to average human behavior, as stored in a lookuptable. The cameraman is, at any given point in time, viewing particularobjects within the object space and his visual axes 432, 434 are shownat such a moment in FIG. 16 directed so as to intersect at a point 436on such an object 438. There will generally be a source of illuminationin the object space (not shown) to which the cameras 428, 430 will besensitive and which will illuminate the objects in the object space,including the point 436, so as to reflect light to the cameras as shown,for example, by light rays or image signals 440, 442. The imageinformation borne by such light rays is encoded in the cameras either byelectromechanically rotated optical means (see, e.g., U.S. Pat. No.3,953,111) or electronically by a suitable nonuniform resolutionencodement method such as one of the scanning methods disclosed above.There will of course be a very large number of reflected light rays,other than rays 440, 442 entering each of the cameras from the variouspoints and objects within the object space. If, for example, the camerasof FIG. 16 use electronic raster scanning of a light sensitive surfaceas the encodement technique, there will be a small number of rays in thevicinity of each of the rays 440, 442 which will be cast oncorrespondingly small areas on each of the respective light sensitivesurfaces. These bundles of rays correspond to a field of view of a fewseconds of arc, from the point of view of the cameraman along lines 432,434. Both of these small areas will be scanned finely in the respectivecameras while all of the remainder of the light sensitive surfaces,excited by the remainder of the rays surrounding the small bundles fromall points within the cameraman's field of view, will be scannedcoarsely. It should be understood that some embodiments (not onlystereoscopic embodiments) may have more than just a few seconds of arcscanned finely. Some embodiments may finely scan on the order of minutesor even degrees. In any event, the control signals 424, 426 will controlthe instantaneous positioning of the finely scanned areas on therespective light sensitive surfaces, according to changes in thedirections of the visual axes 432, 434 of the cameraman's eyes. Forexample, each of the light sensitive surfaces may be laid out in aCartesian coordinate fashion and each of the control signals 424, 426will then contain x and y coordinate information. The optical imagecorresponding to the scene viewed can be thought of as being cast onsuch a coordinate system such that the positive y direction correspondsto “up” in the scene and positive x to “right.” In that case, if thecameraman is gazing at a near object straight ahead and abovehorizontal, then the signal on the line 426 will cause the fine scan inthe left camera to be located in the first quadrant of its lightsensitive surface and the signal on the line 424 will cause the finescan in the right camera to be located in the second quadrant of itslight sensitive surface. A change in the cameraman's gaze toward a nearobject to the far right of the scene below horizontal will cause thefine scan in the left camera to move from quadrant one to quadrant fourand the fine scan in the right camera to move from quadrant two toquadrant four.

Each of the cameras 428, 430 provides a video signal, respectively, onlines 450. 452 to the control units 420, 422, where the imageinformation is formatted, without limitation, into composite videosignals on lines 454, 456, respectively. Of course, other encodementtechniques are equally acceptable, e.g., separate component videosignals for carrying information separately relating to deflection,intensity, etc. A switch 458 is controlled by a signal line 460 from thecontrol unit 422 (or control unit 420) to alternate between signal lines454 and 456 in order to provide each signal alternately on a line 462 toa transmitter 464 for transmission of a broadcast signal on a line 465via an antenna 466.

The timing of the alternations of switch 458 may be selected so as toprovide a left field, frame or portion thereof and then a rightcounterpart.

An antenna 466 a is responsive to the broadcast signal on the line 465and provides a sensed signal on a line 467 to a receiver 468 in an ImageSpace A 470 which provides a received signal on a line 472 to a control474 within an apparatus 26 b. The control 474 strips off synchronizingsignals from the composite video signal on the line 472 and provides oneor more synchronizing signals, e.g., horizontal and vertical, for thedeflection system of an image source 476, as signified by a plurality ofsynchronizing signals on a line 478. The image information is providedon a signal line 480 to the image source 476 which provides alternateleft and right eye images as signified by bundles of image signal lines482, 484, respectively provided to left and right light valves 486, 488.These are in turn controlled by signal lines 490, 492 from the control474. The left light valve 486 is controlled by signal line 490 totransmit images on lines 482 when the left eye images are presented byimage source 476 but to block images on lines 482 when right eye imagesare presented. Similarly, the right light valve 488 is controlled bysignal line 492 to transmit images on lines 484 when the right eyeimages are presented by image source 476 but to block images on lines484 when left eye images are presented. A passive viewer 494 may beresponsive with a left eye 496 to the transmitted left eye images onlines 482 and with a right eye 498 to the transmitted right eye imageson lines 484.

Light valves are not required for an apparatus 26 c in Image Space B 500because two separate image sources 502, 504 provide separate left andright eye images on image lines 506, 508, respectively, providedseparately to a passive viewer's left and right eyes 510, 512. A control514 is responsive to a composite video signal on a line 516 from areceiver 518 supplied by an antenna 520 responsive to a broadcast signalon a line 465 a which may be similar to or identical with the signalbroadcast on the line 465. The control alternately providessynchronizing and left eye image information signals, respectively, onlines 520, 522, and synchronizing and right eye image informationsignals, respectively, on lines 524, 526 to the respective image sources502, 504.

It should be understood that although the apparatus 10 c of Image SpaceB 500 is illustrated as responsive to a single composite video signalwhich is multiplexed between the two separate image sources 502, 504, sothat each source provides images for only half the time, there could aseasily be a system which provides two separate composite video signals,one for each image source, so that each source provides images all thetime. A dual CRT system which requires two separate video signals whichare each always active is shown, for example, in U.S. Pat. No.4,310,849.

Both image spaces are provided with a number of speakers 530, . . . ,532 energized by audio signal lines 531, . . . , 533 in Image Space Aand speakers 534, 536, . . . , 538 in Image Space B energized by audiosignals on lines 535, 537, . . . , 539. These speakers are arrangedabout the head of a passive viewer to simulate sounds “heard” by thesimulated active viewer in the object space. FIG. 16( a) shows thepassive viewer 494 of FIG. 16 having six separate speakers arrangedsymmetrically about his head. Three are arranged in a horizontal circle539 a separated from one another by 120 degrees. The speakers may bemounted, along with the display, in a helmet for mounting on the passiveviewer's head or may be mounted on external supports independent of theviewer. Three others are arranged in a vertical circle 539 b, alsoseparated by 120 degrees. Microphones 540, 542, . . . , 546 aresimilarly arranged about the head of the cameraman in the object space400 and pick up sounds for transmission over signal lines 548, 550, . .. , 552 to the control units 420, 422 for inclusion in the compositevideo signals on the lines 454, 456 in much the same way as conventionaltelevision, except on six channels. These may of course be multiplexedinto a single channel or more.

FIG. 18( a) is an illustration of the manner in which the fields of viewof a simulated pair of eyes in the object space, roughly correspondingto those of the cameras 430, 428 or to those of the cameraman's eyes406, 404 of FIG. 16, are arranged so as to overlap, according to thepresent invention. A left field of view 600 roughly corresponds to thefield of view of camera 430 or the eye 406 while a right field of view602 roughly corresponds to the field of view of camera 428 or the eye404. The median line of a simulated pair of eyes perpendicularlyintersects the origin of the coordinate system illustrated. The leftcamera 430 coarsely scans a field of view roughly corresponding to thefield of view 600 and the right camera 428 coarsely scans a field ofview roughly corresponding to the field of view 602. Depending on theinstantaneous magnitudes of the signals on the lines 418, 416 in theobject space of FIG. 16, each of the respective cameras 430, 428 willselect and then finely scan a small area within the overlapping fieldsof the two cameras. These two finely scanned small areas will themselvesoverlap so as to be in registration, for example, for each successivefixation. For example, if the cameraman's instantaneous gaze is directedto a point of regard to his upper left, then the left camera will scan,e.g., a small area 604 a in the top center of the field of view 600while the right camera 428 will scan a small area 604 b to the top leftof field of view 602 as shown in FIGS. 18( c) and 18(d), respectively.Although the small area 604 a is shown as slightly larger than the smallarea 604 b in FIG. 18( a), it will be understood that this was done forillustrative purposes only, in order to enable the reader to distinguishthe two overlapping areas in the drawing. Of course, for rapidlychanging saccades it will be understood that the two areas may notalways instantaneously overlap, but will try to “catch up” with oneanother when the “target” is acquired and the eyes come to “rest.”Although FIG. 18( a) shows a binocular field of vision covering about105 degrees in a heart shape or an inverted pear shape, it will beunderstood by those skilled in the art of clinical optics that conjugateversion movements of the eyes are only possible over a range ofapproximately 45 degrees from either side of the primary position. Thisis the binocular field of fixation, i.e., the region of space containingall points which may be fixated by the mobile eyes, assuming the headremains stationary. It will also be understood that the relative size ofthe highly detailed image areas in FIG. 18( a), (c) & (d) with respectto that of the lesser detailed area is much larger than it should be ifone were trying to faithfully imitate the actual relationship betweenthe area covered by the fovea, or even the macula, with respect to therest of the retina since the fovea only takes a few seconds of arc. Itshould be understood that the relative sizes selected for the differentareas is a design choice and is not of particular significance for thepurposes of determining the scope of the claimed invention.

It should also be understood that the highly detailed areas 604 a, 604 bmay move in tandem with their associated lesser detailed areas 600, 602or may move independently thereof. The former approach is a closersimulation of the human visual process. If torsions are simulated, forthe former approach, both the highly detailed and lesser detailed imageareas are rotated in tandem; for the latter, they may be rotated intandem but need not be.

FIG. 18( b) illustrates the horizontal field as seen from above. Thecross-hatched area represents the binocular field of view. The extent ofeach of the monocular fields of view is shown. A pair of simulated eyesare illustrated, one on either side of the median line which, asmentioned above, perpendicularly intersects the origin of the coordinatesystem of FIG. 18( a). It will be observed that the monocular centers ofprojection are to be distinguished from the binocular sighting center.

In life, each of the monocular fields 600, 602 of FIG. 18( a) is boundedby the superior and inferior margins of the orbit, the nose, and on thetemporal side by the projection of the edge of the retina (the oraserrata; this extends furthest forward in the eye of the nasal side).Accordingly, as shown in FIG. 18( b), each monocular field extendshorizontally to about 60 degrees nasally and 100 degrees temporally.

Whenever both foveae are stimulated simultaneously, the stimuli areperceived as having a common origin in space. A similar correspondenceexists between the great multitude of other pairs of retinal receptors,called corresponding points which, when stimulated in binocular vision,also give rise to a sensation subjectively localized at a single point.For a given position of the eyes, the locus of all the object pointswhose images fall on corresponding points is known as a horopter,generally a curved surface. The longitudinal horopter is the line formedby intersection of the horopter with the plane containing the eyes'centers of rotation and the fixation point.

Students of the eye often make use of an imaginary organ called thebinoculus or cyclopean eye as an aid for understanding the projection ofimages in binocular vision. If the longitudinal horopter is made a partof a circle passing through the point of fixation and the eyes' nodalpoints, the nodal point of the cyclopean eye should be placed on thiscircle equidistant from the left and right eyes' nodal points. When apoint on the left retina is stimulated, it is conceived as stimulating apoint on the cyclopean retina at the same distance and in the samedirection from its fovea. The same applies to a point on the rightretina. If the right and left receptors under consideration arecorresponding points, they coincide when transferred to the cyclopeaneye where they are said to give rise to a single percept by projectionthrough the cyclopean nodal point. The positioning and overlapping ofthe monocular fields 600, 602 (each field presented only to one of thepassive viewer's eyes) of FIG. 18( a) and the registration of the highdetail image areas 604 a, 604 b are carried out such that a cyclopeaneye positioned with its primary line perpendicular to the plane of FIG.18( a) and intersecting the origin of the illustrated coordinate systemwould have all of the points within the overlapping portions of thefields 600, 602 perceived as corresponding pairs of points.

To digress for a moment, as an active observer moves his body about inan object space, his head and hence the fields of view will move aboutalong with his body. For example, if the active observer walks straightahead for a period of time his head may be kept erect and remain facingstraight ahead or it may from time to time be turned in any direction.His eyes may at the same time fixate on various objects in the objectspace. Under a circumstance of keeping the head aligned straight aheadwith respect to the shoulders, the image content of the monocular fieldsof view will remain approximately the same, ignoring effects due to therocking motion of walking and the monocular field losses accompanyingextreme versions of the margins of the binocular field of view.

For the case where the head is rotated to one side as the activeobserver moves along, the head movement will change the image content ofthe monocular fields of view. According to the present invention, thereare at least two ways of simulating such movements. First, a pair ofnonlinear lenses such as disclosed by Fischer et al in U.S. Pat. No.3,953,111 may be used in the cameras 428, 430 of FIG. 16, each camera orlens mounted on a two degree of freedom platform for rotation abouthorizontal and vertical axes in a “Listing's plane” (a plane through theeye's center of rotation, perpendicular to the primary line of the eye)for each simulated eye or, in the approximate case, for each camera. Thelenses are fitted with artificial boundaries corresponding to those ofthe orbit and nose which are stationary with respect to the cameraman'shead. In this way, the highly detailed and lesser detailed image areaschange their image content together. They stick together, as in life. Asecond way is to change the image content of the lesser detailed area tosimulate head movements only, while the highly detailed image area isfree to move about independently of the lesser detailed image area tosimulated ductions. This approach can be done wholly electronically, bymoving the position of the highly detailed small scan area within thelesser detailed area.

FIG. 17 is an illustration of an apparatus 10 d, according to thepresent invention, which apparatus may be similar to the apparatus 10 aor 10 b of FIG. 16 except that it contains means for presenting imagesat various apparent distances, as in FIG. 3. A composite video signal ona line 550, which may be similar to either the signal on line 472 inImage Space A or on line 516 in Image Space B of FIG. 16, is provided toa control unit 552, which may be similar to the control 474 or 514 ofFIG. 16. The control unit 552 provides a number of left eye signalsincluding synchronizing and image information signals on a multiplesignal line 554 to a left eye image source 558 (solid lines) and anumber of right eye signals including synchronizing and imageinformation signals on a multiple signal line 556 to a right eye imagesource 560 (also in solid lines). Actually, the image sources 558, 560may be either separate left and right image sources as shown in solidlines forming the rectangular blocks 558, 560 (similar to Image Sources1 & 2 (502, 504) of FIG. 16) or may be a single image source as shown inthe left half of each of the solid boxes 558, 560 by broken lines 562 a,562 b, 564, 566 (similar to the Image Source 476 of FIG. 16) made up ofcombined units 568, 570. In that case, light valves 572, 574 areprovided, as in Image Space A of FIG. 16. These would be controlled bysignals on lines 576, 578 as in signal lines 492, 490 of FIG. 16.

In either event, right and left eye images on image signal lines 580,582 are provided to either a single means 584 or separate left and rightmeans 586, 588 for presenting images at various apparent distances. Thesingle means 584 is controlled by a control signal on a line 590 fromthe control unit 552. The separate means 586, 588 are controlled byseparate control lines 592, 594 from the control means 552.

Also in either event, the images carried by the image signal lines 580,582 are altered into images at various apparent distances, asrepresented by image signal lines 580 a, 582 a, and are presented,respectively, to the left and right eyes of a passive viewer 596.

The viewing of a distant object, e.g., along the visual axis of acyclopean eye in its primary position is simulated, according to thepresent invention, as shown in FIGS. 18( c) & 18(d) by centering thehigh detail image areas 604 a, 604 b at points 650, 652, correspondingto the intersection of the left and right eyes' visual axes at the pointof fixation such that points 650 652 overlap on the origin of thecoordinate system of FIG. 18( a).

The visual apparatus is frequently engaged in the acquisition ofdetailed binocular near object information. The fusion reflex directsthe eyes' visual axes so that a near object of regard is simultaneouslyimaged on both foveae. The closest point in the median plane to whichthe eyes can converge is the near point of convergence which variesamong normal individuals between 40 to 160 mm from the corneal plane.

The angle of convergence (C) for a near object is usually measured inprism diopters according to the approximate relation C=−Q×PD, where Q isthe inverse of the distance (q) of the object (in meters) and PD is theinterocular distance (in cm). (A prism diopter (Δ) is a unit formeasuring deviation. One prism diopter is that strength of prism thatwill deflect a ray of light 1 cm at a distance of 1 meter. Thedeflection is toward the base of the prism. Another commonly used unit,a degree (°), equals about 2Δ). For example, given that q=−250 mm andthe PD is 60 mm, C=4×6=24 diopters. Accommodation is measured indiopters, i.e., the inverse of the distance in meters from the eye. Therange of accommodation is the linear distance from the far point to thenear point. For example, an eye with 8D of accommodation has a nearpoint of −⅛ meter or −125 mm, so its range is from infinity to −125 mm.

FIG. 19 illustrates the theoretical relation between accommodation andconvergence as the object of regard approaches the eye. The requiredconvergence is plotted in prism diopters for two different interoculardistances 654, 656 corresponding to 70 and 60 mm, respectively. Aseparate scale at the top of the graph, to be read directly against theobject distance, gives the necessary accommodation.

Various stereoscopic viewing techniques and devices are known. Theserange from the well known Wheatstone, Brewster and Brewster-Holmesstereoscopes to the lesser known Asher-Law stereoscope. Various types ofSynoptophores (a much modified Wheatstone stereoscope) are also known,as is the variable prism stereoscope (using rotary or Risley prisms).The apparati 10 a, 10 b of FIG. 16 in Image Spaces A & B, respectively,are amenable for use in any of the aforementioned stereoscopic viewingdevices, without limitation, for presenting the monocular fields 600,602 of FIG. 18( a) so as to present corresponding points accurately and,if desired, to preserve a normal relationship between accommodation andconvergence in a passive viewer. In both apparati 10 a & 10 b, thecrosses 650, 652 of FIG. 18( c) & 18(d) are optically lined up with theeyes of the passive viewer in their primary positions, i.e., the lefteye's axis perpendicularly intersecting the plane of FIG. 18( c) atcrosspoint 604 a and the right eye's axis similarly intersectingcrosspoint 652 in FIG. 18( d). The actual physical positioning of thefields 600, 602 on the display surface of source 476 or surfaces ofsources 502, 504 is such that they produce an overlap such as issuggested in FIG. 18( a) and such as would be produced in a cyclopeaneye. For the simple case in which the accommodation of the passiveviewer's eyes is not actively stimulated to change, the normal setup ofone or some combination of the known stereoscopic devices, withoutlimitation, may be used without modification, except as described aboveas necessary to ensure the desired overlap and registration of highlydetailed areas.

For the case where accommodation changes are actively stimulated in thepassive viewer, as shown in FIG. 17, it is in many cases desirable topreserve a normal relationship between accommodation and convergencesuch as one of the two relations shown, without limitation, in FIG. 19.This may be achieved in any of the stereoscopes described, or others,using the basic idea of the Asher-Law stereoscope or variations thereofas taught in connection with FIGS. 20, 21 & 22. The basic idea of theAsher-Law stereoscope is to control the convergence/accommodation ratioby separating the two halves of the stereopair, mounting each half onits own adjustably angled rail and varying their separation to apredetermined degree as the viewing distance, which can be kept the samefor each eye, is altered. It should be understood that although thisprinciple will be described in detail in connection with the stereoscopeof FIG. 20, there are many other approaches for achieving the same end.For example, instead of altering the distance of the stereopair from thepassive viewer's eyes (similar to the approach of FIG. 5( d), a variablemagnification lens (similar to FIGS. 5( a) & (b)) or a Risley (rotary)prism (see FIG. 21) or a variable magnification lens as shown in FIGS.21A-J, among others, could be employed to the same end with eitherlateral displacement of the two halves of the stereopair only orrotatable mirrors to effect lateral displacement of the images thereof,or both. Lateral displacement could also be effected whollyelectronically by providing an oversized matrix display and onlyutilizing a portion of the display surface at any given time. Images maythen be shifted left or right by varying amounts to provide the requiredlateral displacement (see FIG. 22). Similarly, all of the other types ofstereoscopes are readily adaptable to control theaccommodation/convergence ratio and the claims of the present inventionembraces all such techniques for achieving the same end in connectionwith the disclosed means and method for presenting simulated activepercepts for passive perception.

The rotary or Risley prism (known in France as the Cretes and in Germanyas the Herschel prism) referred to above is made up of two plano prismsof equal power mounted almost in contact in a carrier. It producescontinuously variable prism power according to the principle illustratedin FIG. 21. The prisms are shown side by side instead of superimposedfor clarity in both FIGS. 21( a) & (b). In the zero setting (FIG. 21(a)), the prisms have their bases opposed. To obtain a desired prismpower, the prisms are mechanically rotated in opposite directionsthrough an equal angle theta. In the position shown in FIG. 21( b), oneprism has its base up and to the left, while the base of the other isdown and to the left. The vertical components cancel out but thehorizontal ones are additive, giving a resultant (always perpendicularto the zero setting) with its base to the left. If the power of each ofthe single prisms is denoted by P, it can be seen from the diagram thatthe total power of the resultant prism is simply 2Psin(theta). Had theinitial rotations been reversed in direction, the resultant would havebeen the same but with its base to the right.

The known variable prism stereoscope incorporates two Risley prisms, sogeared that equal amounts of base-in or base-out prism can be placedbefore each eye. A total of 60 prism diopters is thus available. Aseptum is usually positioned so as to prevent either eye from seeing theopposite half of the stereogram but, according to the present invention,this may be handled by means of light valves, as described above. Also,to permit a normal relation between accommodation and convergence, abase-out prism should be placed before each eye. Otherwise, instead ofconverging to a point in the plane of the surface of the stereogram,each eye might have to diverge in order to fixate a pair ofcorresponding points. If, for example, the separation of these points is7 cm and the viewing distance is ⅓ m, the total base-out prism requiredis 21 prism diopters.

Referring now to FIG. 20 (not to scale), a passive viewer's eyes 700,702 are separated by an interocular distance 704 and are presented witha stereopair 706, 708, shown in three separate arbitrary positions(a,b,c), each half of the stereopair mounted on an angled rail 710, 712for movement thereon via a pair of sleds 714, 716 controlled by acorresponding pair of control signals 592, 594 from e.g., the control552 of FIG. 17. Each half of the stereopair is presented to acorresponding eye of the passive viewer through a fixed lens, e.g., acentered collimating lens (Brewster's lenticular stereoscope) or adecentered (outwards) sphero-prism (Brewster-Holmes stereoscope), theright eye 700 being presented with the right half 706 and the left eye702 being presented with the left half 708. Each rail coincides with aline drawn from the intersection of the lens plane 720 with a medianline 722 to each focal point 724, 726 on the respective optical axes728, 730.

If the convergence/accommodation ratio is to be kept approximately fixedat its normal value of one half the interocular distance regardless ofthe distance of the stereopair from the lens plane, both right and lefthalves of the stereopair must be imaged on the median line or at equaldistances to the same side of it. Thus, for a position of the stereopairat b where the points (GR,GL) and (HR,HL) are two pairs of correspondingpoints, GR and HR are imaged by the right lens 736 in an image plane 740at points 742, 744, respectively, while GL and HL are imaged by the leftlens 738, at points 743, 744, respectively. Rays are shown graphicallyfrom the optical center of each lens through the two pairs ofcorresponding points and on to the image plane. It will be noted thatthe normal convergence/accommodation ratio is preserved for the fixationpoint 744 while it will be correct also for a fixation point 746 ifcertain other conditions relating to the verisimilitude of perspectiveare met. I.e., the correct angular relationship may be maintained bymaking the lateral separation of the cameras equal to that of thepassive viewer's interocular distance and ensuring that themagnification (m) of the images satisfy the relation m=focal length ofstereoscope lens/focal length of camera lens. A third position (c) ofthe stereopair is shown for the case of a distant object.

In further accord with this aspect of the present invention, the railsof FIG. 20 may be eliminated by using a variable magnification lens inconjunction with either an oversized matrix such as is shown in FIG. 22or a pair of base-out Risley prisms. By simply shifting the positions ofthe left and right fields of view on the oversized matrix as shown bythe solid lined fields shifted in tandem to the dashed lined positionsin FIG. 22 or by varying the deflection power of a pair of Risley prismsin tandem, the Asher-Law accommodation-convergence preservative effectmay be achieved using more convenient means.

As mentioned previously, the present invention may be used in a widevariety of embodiments, but generally the embodiments will fall withinone or the other of the classes of either single viewer embodiments ormultiple viewer embodiments.

A single viewer embodiment is shown in FIG. 23 wherein a passive viewer800 is illustrated wearing a helmet 802 having at least one image source804 mounted therein. Additionally, in accordance with the third aspectof the present invention, a plurality of audio sources 806, 808, 810 aremounted at various points in an x-z plane such as shown in FIG. 16( a),and a second plurality of audio sources 812, 814, 816 are mounted atvarious points in an x-y plane such as also shown in FIG. 16( a) so asto achieve the desired all-around audio effect. The helmet in theembodiment of FIG. 23 need not necessarily have the form shown but maytake on a wide variety of forms including the form suggested in U.S.Pat. No. 4,636,866, without limitation. In that case it might benecessary to confine the speakers to a hemisphere situated above the x-yplane of FIG. 16( a). It is even conceivable that a display according tothe invention could be mounted in a pair of goggles. In that case,without limitation to that case, the speakers may be separate from thegoggles such as in the form of a separate headset or speakers mounted ata point distant from the passive viewer, so as to allow others to listenalso, as in a theater environment.

Camera embodiments (not shown) may take corresponding forms includingone which utilizes a helmet with two miniature cameras mounted in or onthe helmet with a corresponding plurality of microphones similarlypositioned about the cameraman's head for picking up sounds in adirectional manner from a plurality of directions, according to thedesign of choice.

Two alternative embodiments of the single viewer helmet of FIG. 23 areshown in FIGS. 24 & 25. These correspond, respectively, to the singleand double image source embodiments shown in image spaces A & B of FIG.16 and are suggested by FIGS. 3 & 4 of U.S. Pat. No. 4,636,866(Hattori). In FIG. 24, a light transmissive liquid crystal display 820is mounted in a helmet 802 a and provides images to a prism 822 whichtransmits a 50% reduced intensity image to each of two reflectivemirrors 824, 826 for providing the reduced intensity images to each of apassive viewer's eyes 828, 830, respectively. Each of the two groups ofreduced intensity images passes through a separate three layer“sandwich” comprising a light valve 832, 834, a variable magnificationlens 836, 838, and a Risley prism 840, 842. Of course, the two groups ofsandwiched elements 832, 836, 840 and 834, 838, 842 need not besandwiched to achieve the desired effect, nor need they be interposed inthe light path in the manner or order shown. In fact, for simplerembodiments, the Risley prisms and even the variable magnificationlenses may be omitted. Electronic controllers 844, 846 correspond to thecontrol 474 in image space A of FIG. 16.

The embodiment of FIG. 25 is similar to that of FIG. 24 except that itcorresponds to the dual image source model shown in image space B ofFIG. 16. It has a pair of transmissive LCD image sources 850, 852, apair of variable magnification lenses 854, 856, and a pair of Risleyprisms 858, 860 separately sandwiched, one sandwich for each eye. Asbefore, the components 850, 854, 858 and 852, 856, 860 need not besandwiched, nor need they be interposed in the exact order shown. And,for simpler embodiments, the variable magnification lenses and theRisley prisms may be omitted. In that case, the embodiment of FIG. 25would be similar to that of FIG. 4 of Hattori, including the fixedmagnification lens 184, except for the nonuniform resolution imageaspect of the present invention. A similar case would exist with respectto FIG. 24 herein and FIG. 2 of Hattori.

FIGS. 26-35 illustrate still another variable magnification lens,according to the present invention. FIG. 25 shows a lens 870 having acentral thin lens section 872 which is flexible and which may, forexample, have a radius of 5 mm. The lens 872 may be made of anethylene-propylene-diene terpolymer vulcanizate such as disclosed inU.S. Pat. No. 4,603,158. The central thin lens section 872 may bestretched thinner by means of a plurality of legs 874 which radiate outfrom the edge of the central thin lens the outer extent of whichdescribe a circle having a radius, for example, of 15 mm. The legs maybe integral to the central lens and made of the same material or may beattached thereto and may be of different material. The central thin lens872 is shown in FIG. 27 in a relaxed state with maximum thickness, forexample of 2.6 mm and with a leg thickness of 0.825 mm. FIG. 28 shows aperspective view. Of course, other means may be used to stretch thelens. The plural leg method is merely one such means.

FIG. 29 shows a plate for mounting the lens 870 on pins 882 as shown inFIG. 30. Through holes 884 at the end of each leg are placed over thepins. A plurality of slots 884 in the plate are used to pass fasteningmeans for attaching the plate 880 to means for stretching the legs suchas shown in FIGS. 31 and 32 while at the same time allowing rotationalmotion therebetween. FIG. 31 shows a plate 890, which may be machined orformed as desired, e.g., from one piece of metal. The plate 890 may beused as a base plate and the plate 880 as a rotating plate which mayrest on a plurality of hook-shaped bosses 892, some 894 of which mayhave holes 896 therein for receiving the fastening means that passthrough the slots 884. Each of the hooked bosses has a post 898associated therewith for mounting a roller. A plurality of such rollers900 are shown in FIG. 32. The rotating plate 880 is mounted on thestationary plate 890 as shown in FIG. 33 and the legs 874 are threadedin between the bosses for rolling contact with the rollers. FIGS. 34 and35 show perspective and plan views of an assembly including a pluralityof screws 902 and washers 904 that hold the rotating plate to the baseplate. The screws pass through the slots 884 and may be screwed intothreaded holes 896 in the bosses. Bushings may be provided on the screwsthat allow the screws to be tightened but that won't clamp down on therotating plate. The rotating plate may be actuated by a linear actuatorhaving a reciprocating piston 908 attached with a sliding coupling to aslot 910 in the rotating plate. The actuator may be mounted to thebaseplate and may have its position controlled by signal lines 912, 914.

For the embodiment shown, the legs 874 are separated by 36 degrees andthe maximum rotation of the rotating plate 880 is about 25 degrees. Thiswill allow the central lens 872 to be stretched from its relaxedposition to achieve a uniform thinning or flattening of the lens wherebyits radius is extended anywhere from 5 to 7 mm by the action of theactuator causing the rotating plate to rotate on the stationarybaseplate thereby causing the legs to roll on the rollers as they apulled and stretched in such a way as to uniformly pull on the lens edgefrom all directions. It will of course be realized that the numericalvalues given for measurements are merely by way of example. It shouldalso be realized that many other ways can be used to stretch such anoptically clear elastomer, according to the teachings hereof.

The second class of embodiment for the present invention is thatincluding embodiments showing images for more than one passive viewer atonce. These will not be described in detail except to suggestembodiments along the lines suggested by U.S. Pat. Nos. 4,427,274 (Pundet al) and 4,514,347 (Reed) and PCT application WO 86/01310 except usingTV projection for projecting hemispherical stereograms and except havingthe passive viewers wearing special stereoscopic effect glasses such as,without limitation, those shown in U.S. Pat. No. 4,424,529. The type ofglasses used depends entirely on the nature of the image, as generallyknown in the arts, for example, of polarized or color differentiatedstereograms. The passive viewers are situated relatively close togetherin a small area or volume confined to a size roughly corresponding tothe cockpit 6 shown in U.S. Pat. No. 4,515,450 (Arrazola).

Although the invention has been shown with stereopairs using the samesurface and using different surfaces separated horizontally, it will beunderstood that presentations of stereopairs in other ways, such as aresuggested in U.S. Pat. Nos. 4,429,328 (Jones, Jr., et al) and 4,523,226(Lipton et al) are within the scope of the present invention.

Similarly, although the invention has been shown and described withrespect to a best mode embodiment thereof, it should be understood bythose skilled in the art that various changes, omissions and deletionsin the form and detail of the foregoing may be made therein withoutdeparting from the spirit and scope of the invention.

APPENDIX

The human visual apparatus comprises a pair of eyes in the form ofglobes, each of about 24 mm in diameter. Each may be thought of as asignal converter, similar to a video camera, having means responsive tolight rays or signals in an object space for conversion by means of alens to optical images or image signals, each globe having a lightsensitive surface responsive to the optical image signals for conversionto electrical image signals at an output thereof.

Emerging from the white at the eye's entrance is the transparent tissueof the cornea for bending incoming light through a clear fluid calledthe aqueous humor and through an opening in the iris called the pupil.The light then passes through the crystalline lens, which focuses anoptical image cast on the retina. Before reaching the retina the lightpasses through the vitreous humor, a jellylike substance which more orless prevents further bending of the light as it exits the lens. Thepupil changes in size according, among other causes, to changes inluminance, in order to regulate the intensity of the light which is caston the retina.

The cornea is a meniscus structure with an anterior radius of curvatureof 7.7 mm and a posterior curvature of 6.8 mm. It bulges slightly upwardfrom the surface of the white of the eye, like a watch crystal. It isabout 12 mm in diameter and 0.5-0.6 mm thick at the center. It is highlytransparent and has an index of refraction of 1.376. It causes the firstand largest bending or refraction of all of the eye's refractingelements. The light rays emerging from the back of the cornea have thusbeen bent sharply inward toward each other.

In this more tightly clustered condition they emerge into an anteriorchamber filled with a colorless liquid (again, the aqueous humor) lyingbehind the cornea and in front of the iris and crystalline lens. Asmentioned, the aqueous humor is virtually colorless and is closelymatched to the cornea in refractive power. Thus, the direction of theconverging rays are essentially unchanged in the aqueous humor. Thedistance from the posterior surface of the cornea to the crystallinelens is about 3 mm.

The crystalline lens provides a certain degree of refractive power forthe eye and also provides the mechanism for focusing at variousdistances, called accommodation. The lens is biconvex, has a diameter ofabout 9 mm and has a thickness at the center of about 3.6 mm in theunaccommodated state. It is highly elastic and is suspended from itsperiphery by ligaments attached to a ciliary body for controlling thecurvature of its surfaces by means of the ciliary muscle varying thetension on the ligaments. The radii of curvature of the anterior andposterior surfaces are, respectively, 10 mm and −6 mm, for theunaccommodated state (ciliary muscle relaxed, suspensory ligaments attheir greatest tautness and the lens surfaces assuming their flattestcurvatures) for viewing far objects. With the introduction ofaccommodation, both lens surfaces, but especially the anterior, becomemore steeply curved and the thickness increases for viewing nearobjects. As mentioned above, the posterior surface of the lens is incontact with the vitreous humor, a transparent gel which fills theposterior of the globe. The index of refraction of the vitreous humor isabout the same as the aqueous humor. The index of refraction of the lensis nonuniform, being higher (about 1.40) near the nucleus and lower(about 1.375) near the outer surfaces, resulting in a power greater thanit would have if its refractive index were uniform.

The retina is anatomically an outgrowth of the brain and forms a thin,but intricately structured lining of the posterior portion of the globe.Optically speaking, the retina is akin to a spherical projection screen(radius of curvature of about −12 mm) upon which optical images arecast. Retinal receptors are stimulated by light and transmit impulsesacross the retinal surface via nerve fibers and exit the eye via theoptic nerve trunk on their way to the cortex. There is of course anorderly arrangement of receptors in the retina, together with itsconnections to the cortex. The retina's ability to sense image detail,however, is nonuniform over its surface and reaches a maximum in themacular region. This is approximately circular area of about 1.5 mmcontaining a smaller central area, the fovea centralis (about 0.3 mmhorizontally by 0.2 mm vertically), populated exclusively by retinalcones. It is at the fovea that the eye attains its maximum resolvingpower. When an object engages visual attention, the eye is instinctivelyturned so that the image lies on the fovea. The optical axis of the eyedoes not, as might be expected, exactly intersect the fovea. A “visualaxis,” distinct from the optical axis is therefore postulated ascoinciding with the chief ray of the pencil of rays which enters thepupil and is converged to the fovea. The visual axis is normallydisplaced nasally about 5 degrees and upwardly about 2 degrees from theoptical axis. Although the axial terms “optical” and “visual” aredistinct, they are used interchangably herein.

From the foregoing it will be understood that parts of the eye, i.e.,the cornea, aqueous humor, iris, lens and vitreous humor, cooperate toform optical images or “image signals” which are cast on the retina fornonuniform detail conversion or encodement to electrical signal impulsesfor transmission to the brain via the optic nerve. There, percepts ofthe object space represented by the optical images are formed.

The monocular field of view of each eye extends horizontally throughmore than 90 degrees from the optical axis on the temporal side. Thenose, brow and cheek limit the monocular field of view in otherdirections, so that its shape is irregular.

The primary position of each eye is looking straight ahead at a distantobject with head and shoulders erect. The eye is approximately sphericaland its movements are akin to those of a ball and socket joint.Rotations of an eye from the primary position are called ductions.Ductions are defined as rotations about either a horizontal or verticalaxis in a vertical plane in the head passing through the center ofrotation of each eye and normal to the visual axis of the eye in itsprimary position. Ductions thus represent secondary eye positions. Thevisual axis, the horizontal axis and the vertical axis intersect at thecenter of rotation. Elevation of the visual axis is called supraductionand moves the cornea upwards. Depression or infraduction moves thecornea downwards. Abduction and adduction are, respectively, movement ofan eye away from and toward the nose. A tertiary eye position is acombination of a horizontal and a vertical rotation and results in anoblique direction of gaze, for example, up and to the left.

Version is movement of both eyes in a similar direction, e.g., whilemaintaining binocular fixation on an object moving in a fronto-parallelplane (a vertical plane parallel to the vertical plane in the head,described above). Dextroversion is a movement of the subject's right eyeaway from the nose (abduction) and left eye toward the nose (adduction).Laevoversion id adduction of the right eye and abduction of the left.Elevation of both eyes is called supraversion and depressioninfraversion.

Binocular vision is the use of two eyes in such a coordinated manner asto produce a unified mental percept of an object space. The cerebralcortex receives separate bundles of “encoded” image signals or “neuralimages” from each eye in response to separate images from slightlydifferent perspectives cast on the separate retinae. A simplesuperposition of the two images would give rise to double vision and aconflicting sense of direction. The mental percept or “cotical image” isthe result of the blending or fusing of the two neural images orrepresentations in the higher levels of the brain—the psychologicalstage of the visula process. The two monocular impressions must bebrought into a corresponding association in the cortex and the brainmust be capable of fusing or integrating them into a single binocular“picture.” Stereopsis is thus the fusing of these “neural images” intothree-dimensional percepts of an object space as a result of the slightdifference in perspective between the optical images cast on the leftand right retinae. It will of course be understood that there are notrue pictures in the cortex.

The field of view is the extent of an object space containing all pointswhich produce perception by way of the stationary eye, provided thestimulus is sufficient. For binocular vision to be possible, the twoorbits and the structure of the eyes must be arranged so that the visualfields overlap. Happily, the orbits are positioned in front of the skulland although their axes diverge at about forty-five degrees, the eyesare nevertheless mounted in their respective orbits so that their visualaxes are approximately parallel.

The monocular field of view is bounded by the superior and inferiormargins of the orbit, the nose, and on the temporal side by theprojection of the edge of the retina. The field of view extends to aboutsixty degrees nasally and one hundred degrees temporally. The overlap ofthe two monocular fields is the binocular field of view and may be seenby plotting each on the same chart; the area of overlap is approximatelythe shape of an inverted pear. The monocular temporal fields, however,contribute a great deal to spatial perception.

The field of fixation is that region of space containing all pointswhich may be fixated by the mobile eye, the head remaining stationary.Conjugate versions over a range of about forty-five degrees from theprimary position are possible. It is important to distinguish betweenthe fields of view and fixation as the first relates to the stationaryeye(s) and the second to the motor field—the solid angle within whichthe conjugate visual axes can be moved. In life, the visual field iseffectively increased by both head and eye movements. An object in theperipheral field catches our attention and the eyes move so that theimages fall on the foveae. Coordinated response of head and eyemovements is required, the eye movements themselves rarely exceedingtwenty degrees.

Eye movements are capable of high angular velocity, up to nine hundreddegrees per second, for acquiring image information from peripheralareas to the center of vision, at the fovea, for detailed examination(saccades). Saccades may occur three or four times per second, eachlasting about 20-30 ms. Successive saccades are spaced by at least 150ms, since it takes about 50 ms to “program” the next one during afixation, 20-30 ms to execute, 50 ms to regain clear acuity and aminimum of 50 ms to acquire a new visual scene for assessment,interpretation and integration with previous scenes. Once acquired, atarget may be held at the fovea when the scene or viewer is in motion(pursuit). Clear vision may also be maintained with approaching orreceding targets.

For each of the distances viewable by a normal eye between infinity andthe nearest point of distinct vision there will be a corresponding stateof accommodation. For binocular vision, it is necessary to considerconvergence as well as accommodation since these two actions arenormally associated.

Convergence is the power of directing the visual axes of the two eyes toa near point as opposed to a distant object where accommodation is atrest and the visual axes are parallel. When an observer views a nearobject, he is compelled both to accommodate and to converge for thatdistance; with a certain amount of accommodation, a corresponding effortof convergence of the visual axes is associated; the relationshipbetween accommodation and convergence is harmonious but notunchangeable.

1. Apparatus, comprising: means for providing successive mixed opticalimages of an object space in an image space for presentation to aviewer's eye, each mixed optical image having a highly detailedcomponent and a lesser detailed component, the successive mixed imagesfor simulating percepts of optical images of the object space cast on asimulated eye's retina; and means for changing the image content ofselected successive mixed images according to changes in the directionof the simulated eye's visual axis in the object space such that therelative direction of the visual axis of the viewer's eye with respectto the image space may analogously and passively follow the relativedirection of the simulated eye's visual axis with respect to the objectspace by following the highly detailed component of the successive mixedoptical images presented thereto such that the highly detailed componentmay be cast on the fovea of the retina of the viewer's eye and thelesser detailed component may be cast on at least the remainder of theretina of the viewer's eye for stimulating the viewer to experiencepercepts corresponding to the simulated percepts.
 2. The apparatus ofclaim 1, wherein the highly detailed component is mobile with respect tothe lesser detailed component to simulate movement of the simulated eyewith respect to the simulated eye's orbit, the lesser detailed componentencompassing a field of view which changes to simulate movement of theorbit's head with respect to the object space.